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Abstract 


This technical document provides guidance to educators on the creation and interpretation of 
survey instruments, particularly as they relate to an analysis of program implementation. 
Illustrative examples are drawn from a survey of educators related to the use of the easyCBM 
learning system. This document includes specific sections on considerations related to sampling 
plans; instrument creation; and gathering, analyzing, and reporting of survey results. 
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SLDS Technical Document #4: 

Instrument and Survey Analysis Technical Report: Program Implementation Survey 

We begin this technical document with a brief definition of key terms and then provide 
more in-depth discussion of considerations related to sampling plans; instrument creation; and 
gathering, analyzing, and reporting of survey results. 

Key Terms 

Please note: These terms are defined in relation to surveys, here, but they may also have 
additional meaning when used in different contexts. Definitions provided here are not intended to 
be exhaustive. 

Instrument: A testing device for measuring a given phenomenon. Instruments may include 
paper/pencil or computer-based tests, interviews, and surveys, among others. 

Survey: A data collection tool used to gather infonnation from individuals. 

Respondent: A person responding to a survey. 

Population: The group of people whose input on a topic one is trying to obtain through 

surveying. If the population is small enough, it might be possible to include everyone in 
the population in one’s sample. Otherwise, one uses a sampling plan to identify in 
advance the people who will be included in the sample, and how they will be selected for 
inclusion. 

Sample: The group of people to whom the survey is sent. 

Selected Response Item: A question where respondents select their response from a pre- 
determined set of possible answer choices (aka “multiple choice”). 

Constructed Response Item: A question where respondents answer a question using their own 
wording, typically in a short-answer or essay-type fonnat (aka “open ended question”). 
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Considerations in Sampling 

One of the first decisions one must make before conducting a survey is whom to include. 
Typically, this is done through the creation of a written sampling plan, whereby one specifies the 
population of interest and describes whether everyone in the population will be included 
{exhaustive or census sampling ) or a smaller sub-set of the population. In schools, it is not 
uncommon to solicit input from every member of the population of interest. In large districts, 
however, the sheer number of potential respondents may make a smaller sample more 
manageable. Knowing in advance how the results will be compiled and analyzed can help in 
making decisions about sampling plan. For instance, if one is going to be using a paper/pencil 
survey and hand-entering all the results prior to analysis, then a smaller, more selective, planned 
sample may be in order. In contrast, if one is using one of the many online survey tools where 
the results are tabulated by the computer as they come in, a larger sample may be feasible. 

If one decides to sample (rather than send a survey to every member of the population in 
question), the next decision is what sampling method to use. Generally speaking, sampling 
methods can take either a probability or nonprobability approach. 

Probability Sampling. In probability sampling, each member of the population has a 
chance of being selected, and one either samples randomly or systematically. In a random sample, 
each member of the population has an equal opportunity to be selected. In a systematic sample, 
one selects people to include in the survey using a systematic, planned approach. For instance, 
one might select every n th name to survey. One danger of this approach, of course, is that names 
are not randomly distributed across the population. Another systematic sampling technique 
stratified sampling first requires one to identify relevant clusters, or stratifications, into which 
members of the population might be grouped. Once these groups are identified, and the 



Instrument & Survey Analysis 


Page 3 


population is organized by the stratifications, one selects a sufficient number of people from each 
of the groups to enable generalization to the population in question. In a school district this 
approach might be taken, for instance, if educators wanted to ensure that they had attempted to 
include input from a representative sample of people from each ethnic group enrolled in the 
school. 

Nonprab ability sampling. Nonprobability sampling involves selecting people to survey 
in a nonrandom manner. Often, one will see such sampling plans referred to as convenience, 
judgment, or snowball sampling. As the name implies, convenience sampling refers to a situation 
whereby people are selected to participate in the survey because they are convenient. For 
example, a school might conduct a survey of parents in attendance at back-to-school night using 
convenience sampling. One danger of convenience sampling, of course, is that the people 
included may not truly represent the larger population to which one is hoping to generalize. In 
the example of sampling parents at a back-to-school night, for instance, this approach might 
result in an inability to gather input from parents who work nights. To the extent that their views 
might differ from the views of the parents in attendance at the evening event, the sample results 
would be biased. Judgment sampling is typically an extension of convenience sampling whereby 
participants are selected based on the judgment of the person conducting the survey that they will 
be a good representation of the population as a whole. For instance, a school might opt to survey 
students in one classroom per grade, selecting those classrooms that appear to best represent the 
school population as a whole. The biggest concern with this approach, of course, is the degree to 
which the sample was selected wisely. To the extent that the sample differs from the population 
as a whole, the results may not be generalizable. 
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A final survey technique, most often used with hard-to-identify samples, is called 
snowball sampling. In this approach, one uses referrals from others to identify and recruit 
additional people to include in the sample. For instance, a school district may want to solicit 
information about how they might better provide assistance and support to homeless students and 
their families, but it is unlikely that school records will enable the easy and accurate 
identification of which parents should be surveyed. Using a snowball sampling technique, one 
would identify one or more homeless family in the district and would then request that person’s 
assistance in passing along the survey to other families in similar circumstances. A caution when 
using this approach is that the final sample may well be biased, as the people surveyed may end 
up sharing characteristics, through their associations with one another, other than the one that 
originally led to their identification. 

What to Look for in Reference to Sampling Design in Research. Because the approach 
used to gather the sample can impact the generalizability of the results considerably, it is 
important to examine the sampling plan and sample characteristics carefully when reading the 
results of a survey. Things to notice include how the sample was generated, the demographic 
characteristics of the sample (especially as they compare to your own population of interest), and 
the size of the sample compared to the identified population. Generally speaking, the closer the 
sample is in size and characteristics to the population as a whole, the more confident one can be 
in the results being reported. 

Considerations in Instrument Creation 

Surveys vary widely in their construction and intended use. Knowing in advance what 
one will do with the results can help ensure that the survey is useful and focused (Fowler, 2009). 
Keep in mind that if the survey is too long, it is likely that fewer people will complete it. 
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Generally speaking, responses to selected-response items are easier to analyze (many of the 
online survey tools provide a way to download results directly to an Excel file). Oftentimes, 
though, one may want to include constructed-response items or places where respondents may 
add more detail to explain their responses. Such responses can provide interesting insight into 
patterns in the results. It is important to keep in mind, however, the potential that a well-crafted 
constructed response might unduly sway one’s interpretation of the results. 

Creating selected-response items. In designing a survey, it is important to keep in mind 
that well- written questions will help respondents provide accurate and reliable answers. If your 
survey includes questions related to a sensitive topic, it may be wise to ensure that responses 
remain anonymous and to make sure that respondents are aware of this before they begin the 
survey. To avoid the situation where people select the ‘safe’ middle option choice rather than 
taking a stand on a topic with a more definitive answer, it is generally recommended to have an 
even number of response options when asking about people’s opinions on a topic. The following 
guidelines can help ensure that questions solicit useful responses: 


Table 1 


Selected-Response Item Writing Guidelines 


Guideline 

Example of a Poorly- Written 
Question 

Example of a Better- Written 
Question 

Limit each question to one 

How fair and appropriate are the 

How appropriate are the tests in this 

dimension 

tests in this class? 

class: 


a) extremely 

a) extremely 


b) somewhat 

b) somewhat 


c) not very 

c) not very 


d) not at all 

d) not at all 
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Table 1 


Selected-Response Item Writing Guidelines (Continued) 


Minimize the repetition across 
answer choices ~ include words that 
might repeat in the question stem so 
they do not need to be included in 
the responses. 

How would you describe the 
campus? 

a) I feel very safe on campus. 

b) I feel somewhat safe on 
campus. 

c) I do not feel very safe on 
campus 

d) I do not feel safe at all on 
campus. 

How safe do you feel on campus? 

a) very 

b) somewhat 

c) not very 

d) not at all 

Keep the grammar structure parallel 
in each of the possible response 
choices. 

How useful are the student behavior 
reports during parent conferences? 

a) best ever! 

b) they are quite useful 

c) somewhat useful 

d) not at all 

How useful are the student behavior 
reports during parent conferences? 

a) extremely 

b) quite 

c) somewhat 

d) not at all 

Make sure that the answer choices 
cover all possible responses. If 
people might feasibly be able to 
select two options, be sure they are 
given that choice. 

Which class do you most enjoy? 
(select one) 

a) English 

b) Mathematics 

c) Spanish 

d) PE 

Which class(es) do you most enjoy? 
(select all that apply) 

a) English 

b) Mathematics 

c) Spanish 

d) PE 

e) Other (explain) 

Make sure that the answer choices 
are mutually exclusive. If they are 
not, then respondents might not 
know how to give an accurate 
answer. 

Where do you usually do your 
homework? 

a) In the kitchen 

b) In front of the TV 

c) In my bedroom 

Where do you usually do your 
homework? 

a) In a quiet place at home 

b) In a noisy place at home 

c) In a quiet place away from 
home 

d) In a noisy place away from 
home 

e) Other (explain) 
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Table 1 

Selected-Response Item Writing Guidelines (Continued) 

Ensure that the questions flow 
smoothly from one to the next. 
Grouping questions that are similar 
can help make it easier for 
respondents to complete the survey. 

How long have you lived in the 
district? 

a) >5 years 

b) < or = 5 years 

What is your GPA 

a) 4.0 -3.5 

b) 3.49-3.0 

c) <3.0 

Where did you live before you lived 
here? 

a) in a neighboring district 

b) in a different state 

c) NA: I have always lived 
here 

How long have you lived in the 
district? 

a) >5 years 

b) < or = 5 years 

Where did you live before you lived 
here? 

a) in a neighboring district 

b) in a different state 

NA: I have always lived here 
What is your GPA 

a) 4.0 3.5 

b) 3.49-3.0 

c) <3.0 

If there is a chance that a respondent 
will not be able to answer a 
question, include a “don’t know” 
answer option. Otherwise, 
respondents may guess, reducing the 
reliability of your results. 

What proportion of your waking 
hours do you typically spend 
reading? 

a) Less than 25% 

b) More than 25% 

What proportion of your waking 
hours do you typically spend 
reading? 

a) Less than 25% 

b) More than 25% 

c) I don’t know 

Write questions and responses in 
such a way that they do not “lead” to 
a particular response. 

Do you agree that reading for 
pleasure outside of school is 
extremely important? 

a) Yes 

b) No 

c) I’m not sure 

How important is reading for 
pleasure outside of school? 

a) Extremely 

b) Somewhat 

c) Not very 

d) Not at all 

Avoid the use of unfamiliar words, 
abbreviations, and technical jargon. 

How satisfied are you with the 
district’s efforts to support work 
toward AYP? 

a) Extremely 

b) Somewhat 

c) Not very 

d) Not at all 

How satisfied are you with the 
district’s efforts to support 
improvement in student performance 
on statewide large-scale 
assessments? 

a) Extremely 

b) Somewhat 

c) Not at all 

d) I’m not sure 
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Table 1 


Selected-Response Item Writing Guidelines (Continued) 


Limit questions that ask respondents 

Rank the following in their order of 

Rank the following characteristics of 

to rank order a series of items to no 

importance: (1 = most important; 10 

cafeteria food in their order of 

more than 5 items. 

= least important) 

importance: (1 = most important; 5 = 


a) healthy cafeteria food 

least important) 


b) tasty cafeteria food 

a) healthy 


c) inexpensive cafeteria food 

b) tasty 


d) a variety of cafeteria food 

c) inexpensive 


e) large servings of cafeteria 

d) a variety 


food 

f) easily-recognizable 
cafeteria food 

g) cthnically-diverse cafeteria 
food 

h) warm cafeteria food 

i) etc. 

e) large servings 


Creating constructed-response items . A great deal of attention must be paid to the way 
in which constructed-response items are phrased if one is to get a lot of infonnation from them. 
A good rule of thumb to remember is that the best constructed-response questions will solicit 
more elaborate responses. Avoid phrasing a question in a way that might lead a respondent to 
answer with a single word response (eg., “yes” or “no”). Phrasing a question the same way you 
might if you were going to be conducting an interview can sometimes help. For instance, asking 
people to Describe the most useful aspects of today ’s professional development workshop may 
elicit more detailed responses than asking people What made today ’s professional development 
workshop useful? Likewise, one might decide to phrase a question in a neutral way that would 
enable respondents to share insights either negative or positive as a means of ascertaining their 
true feelings on a topic. For instance, a question phrased as Describe your opinion of today ’s 
professional development workshop could elicit either negative or positive responses, which 
might provide a more accurate gauge of how people viewed the event than responses that 
specifically solicit feedback about positive aspects of the workshop. 
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What to Look for in Reference to Item Type in Research. Often, survey designers will 
include a mix of selected- and constructed-response items on the survey instrument. It is 
important to remember, however, that not all respondents will have the same ability to express 
themselves well in writing. There is always the danger that a well-written constructed-response 
will skew one’s interpretation of the results as a whole. For this reason, it may be wise to use the 
results of selected-response items to help balance the opinions shared in open-ended answers. 
When reading the results of survey research, look for examples of the questions themselves and 
evaluate them for lack of bias and appropriate phrasing, keeping in mind that poorly- written 
questions reduce the reliability and validity of conclusions drawn from the survey. 
Considerations in Administering Surveys 

Coming up with a sampling plan and creating a survey instrument are critical first steps. 

It is also important, however, to make plans to increase the response rate. The following steps 
may help increase people’s willingness to respond to a survey. 

• If mailing the survey, use an envelope that will help your survey stand out in the mailbox. 
Generally speaking, colored, hand-addressed envelopes with postage stamps affixed 
(rather than bulk mail postage or printed address labels) may help reduce the chance that 
the survey is discarded as “junk mail” prior to being opened. 

• Include a brief but well- written cover letter, introducing the survey and emphasizing the 
importance of receiving a response from the person. If an online survey instrument is 
being used rather than a mailing, a short well-written e-mail takes the place of the printed 
cover letter. 


Include a title for your survey that is short and meaningful. 
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• Include short, easy-to-understand directions for completing and returning the 
survey. If the survey is being distributed through the mail, include the return 
address on the survey itself as well as on the reply envelope. 

• Begin the survey with a few interesting and non-threatening questions, and 
organize the survey such that the most important questions appear within the first 
half of the survey. That way, even if people do not complete the entire survey, the 
responses to the most important questions will be captured. 

• If the survey will be administered paper/pencil, leave plenty of white space. This 
helps give the impression that the survey will be easy to complete. 

• Consider incentives to encourage people to return completed surveys. For mailed 
surveys, attaching a dollar bill may help improve the response rate. Offering to send 
a summary report of the results to everyone who completes the survey may also 
help encourage people to respond. 

• Piloting the survey with a few people who represent the target population prior to 
distributing it more widely can help identify points of confusion to be corrected 
before the survey is administered more formally. 

Online versus Paper / Pencil Surveys. Widely-accessible technology in the fonn of 
internet-based survey tools has opened new avenues for collecting survey data. Online surveys 
may help reduce time and costs associated with gathering and analyzing survey data (Clayton & 
Werking, 1998; Gjestland, 1996; Schmidt, 1997). One of the most appealing characteristics of 
online survey engines is the way in which they facilitate data entry. Each respondent’s input is 
tallied as it is entered, enabling instantaneous download of the data for rapid analysis as soon as 
the survey is closed. They may also include built-in features to enhance response rates, such as 
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the ability to provide feedback and/or summaries of responses upon completion and simple-to- 
use tracking of who has responded, enabling the person conducting the survey to send out 
multiple invitations to participate without an increase in mailing costs (Dillman, 2000; Schmidt, 
1997), although some research suggests that paper/pencil surveys result in higher response rates 
(Sax, Gilmartin, & Bryant, 2003). Online survey engines are not without problems. One of the 
biggest challenges of using online survey engines is sampling bias that might favor higher- 
income households, where computer access might be more readily available (Sax et al., 2003). 

Paper/pencil surveys may have some distinct advantages for school-based applications, 
where potential respondents are likely to be physically present in a room (such as a parent or 
faculty meeting or student class). Distributing, completing, and collecting surveys in the same 
sitting, can improve response rates significantly. Likewise, some respondents may find a 
paper/pencil survey less intimidating to complete than one that is delivered via computer. 
Knowledge of the potential sample to be surveyed can help in selecting the most appropriate 
method to use. 

Analyzing Survey Results 

Survey results are generally reported for the whole sample as well as for specific sub- 
groups. To make such analyses possible, one must collect the data in such a way that it is 
possible to link the responses to the demographic information collected in earlier questions. For 
instance, if one is interested in comparing the opinions of freshmen students with the opinions of 
juniors, the survey instrument must include a question that will prompt students to select their 
class in school so that the responses can later be disaggregated or different surveys must be 
administered to students in the different groups. One of the most common mistakes made by 
people who have less experience with constructing and analyzing surveys is failure to include 
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questions that will enable thorough analysis of the results. Taking steps to ensure that all 
responses on a given survey can be tracked back to the individual responding to the survey does 
not mean breeching confidentiality. One can assign code numbers to each survey instrument and 
use the numbers to link individuals’ responses to different questions. Such an approach is 
particularly important for surveys that extend beyond a single sheet of paper; printing the 
person’s code number on each sheet of paper can help ensure that responses are tracked even if 
the individual pieces of paper become separated. 

Reporting Survey Results 

When reporting the results of a survey, provide a summary of the process used including 
the method of distribution and how the survey was introduced. Include infonnation about how 
surveys were distributed and how many were completed, calculating a response rate (usually 
reported as a %) for the survey as a whole. If some questions elicited significantly fewer 
responses than the overall completion rate for the survey, report this infonnation as well. Provide 
a frequency count, with raw numbers as well as percentage of respondents, for items that use a 
Likert-type scale. Including descriptive statistics (mean, standard deviation, and total number of 
respondents) may also help in interpreting results. Sometimes, other measures of central 
tendency such as which option elicited the most responses (mode) provide useful insight. If the 
survey includes constructed response items, one may either analyze the responses given for 
patterns and report findings with a few illustrative quotations or provide all responses given, 
allowing others to draw their own conclusions. Of these two approaches, the first may help 
reduce the potential for the qualitative responses to overpower the trends captured in the 
quantitative items on the survey. 
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Illustrative Example of Survey Creation, Administration, Analysis, and Reporting 

The section that follows provides an annotated technical report describing a survey 
created in the mid-2000s to collect information about teacher self-efficacy. It is included in its 
entirety, in the same form in which it was published, with one exception: the addition of 
annotations intended to call attention to features in the manuscript discussed in this technical 
document. 
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Abstract 

This technical report describes the development, pilot testing, and revision of a survey 
instrument designed to measure secondary school teachers’ perceptions of their efficacy working 
with students from diverse backgrounds. A brief review of relevant literature frames the current 
study in the context of survey development that is technically adequate. Exploratory factor 
analysis is used to identify factors within each of four hypothesized latent constructs (alignment, 
inclusivity, organization, and general efficacy), and the measurement model fit is evaluated to 
explain their representation. Results suggest that the survey instrument (once revised) is reliable 
and that the measurement model may adequately fit the data. 
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literature which 
forms the 
theoretical / 
conceptual 
framework for this 
work. 


Testing a Measurement Model 


School improvement, as mandated by the federal government and imj 


all state education agencies, is a continuing effort on the part of school staff t( 


achievement on standardized measures (Buttram & Waters, 1997). The past 25 years has seen a 
growing movement toward school improvement throughout the United States. Beginning with A 
Nation at Risk (National Commission on Educational Excellence, 1983), continuing through the 
President’s Education Summit in 1996 (Buttram & Waters, 1997), and, more recently, the No 
Child Left Behind Act of 2001 (2002) significant public resources have been dedicated to 
improving school effectiveness through standards-based accountability. 

In the case of school improvement and accountability, the construct under investigation 
may be improving student achievement, but efforts toward that end generally take the form of 
interventions, not with the students, but with the teachers (Barth, 1990, Rosenholz, 1989). This 
approach reflects the broadly held position that improving student achievement relies on 
changing what happens in the classroom (Rudy & Conrad, 2004). 

Researchers at the University of Oregon have undertaken an effort to identify, and to 
subsequently measure, elements of teacher attitudes and competencies that have an impact on the 
effectiveness of staff development efforts in school improvement. The elements described here 
were identified as consequential to a staff development effort targeting instructional change 
toward concept-based instruction (CBI) as opposed to more common fact-based instruction 
(Carnine, 2002; McCoy & Ketterlin-Geller, 2004; Twyman, Ketterlin-Geller, McCoy, & Tindal, 


2003). Our focus has been in the secondary school content areas. 
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Briefly, concept-based instruction attempts to identify concepts common across large 
portions of the curriculum and to explicitly address these concepts rather than to focus upon the 
facts of individual and independent instances throughout curricula. We focus on four constructs 
of school improvement as reported by teachers in conversation following training and 
implementation of concept-based instruction in the classroom. Teachers participating in staff 
development in the use of concept-based instruction anecdotally report improvement in their 
self-efficacy in these areas and often see associated improvement in student academic 
achievement. The challenge, then, is to identify or create an instrument that can measure 
teachers’ self-efficacy in these areas before and after their involvement in staff development so 
that the change might be more easily identified and its relationship to student academic 
achievement more explicitly described. This technical report describes the survey instrument 
developed to measure these four latent constructs. 

Alignment. As a result of the push toward academic achievement as measured by 
standardized assessments, it is critical to establish alignment among state standards, curriculum, 
instruction, and assessment. To improve student outcomes we must evaluate student performance 
(American Educational Research Association (AERA), 2003; Rudy & Conrad, 2004). Alignment 
of these four constructs provides an assurance, first, that the tests appropriately measure student 
knowledge and, second, that the results of those tests can be used to modify both curriculum and 
instruction to better address the standards and improve student achievement (AERA, 2003). 
Without alignment among these elements of schooling, data may be suspect as school and 
district staff modify their practices to facilitate improvement. Although responsibility for 
aligning statewide testing to state standards falls reasonably to state department of education 
staff (AERA, 2003), aligning the curriculum and associated instruction must occur at the local 
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level and focus on curriculum and instruction to standards and associated assessments. Raising a 
teacher’s perception of efficacy in alignment would enhance student instruction toward the 
standards and therefore toward statewide assessments. 

By aligning the curriculum to standards, teachers can better support the learning of all 
students relative to these standards. Despite efforts at mandating content standards as the basis 
for curriculum-design decisions, textbooks often dictate the curriculum rather than these 
standards (Diegmueller, 1995). Exacerbating matters, the content in these textbooks typically is 
not aligned to the standards, nor do textbooks present information on an intellectual level called 
for by state standards (Tomlinson, 2000; Ketterlin-Geller, McCoy, Twyman, & Tindal, 2003). 

Inclusivity. The No Child Left Behind Act of 2001 (NCLB) (2002) presents schools with 
the challenge of helping all students, regardless of special needs status, to meet the same high 
standards of academic performance. This challenge has been educators’ concern for some time 
and is reflected in the trend toward inclusion of special needs students in regular education 
classrooms (Alper, Schloss, Etscheidt, & Macfarlane, 1995). Inclusion is intended to provide 
students with a functional instructional environment that is based on their needs (e.g. classified 
special needs, at risk, homeless, or gifted) (Stainback, Stainback, & Jackson, 1992). 
Unfortunately, teachers not yet comfortable with their inclusive classroom now are confronted 
with the demands of NCLB. 

Inclusivity, then, is influenced by teacher’s propensity to manage their classrooms in an 
encompassing manner based on curriculum adaptation and instructional scaffolds rather than 
mere presence of students from various groups (like disabilities and second language). Some 
might argue that unlike placement, inclusivity cannot be mandated but must be the result of a 
change in teacher attitude and behavior. Corbett (2001) argues that past practices in inclusion 



Teacher Efficacy Survey 7 


that relied simply on placement in the general education classroom in defining success are 
outdated and inadequate. This attitude of “dump and hope” (Corbett, 2001, p 58) cannot, 
however, be replaced by a more openly inclusive instructional model without the full 
participation of the classroom teacher. A change in the teacher’s perception of self-efficacy in 
addressing the needs of a broader spectrum of student abilities seems critical to development of 
an attitude of inclusivity. 

Organization. Such focus on teachers is unlikely to be productive without concurrent 
consideration of the classroom and school demands. The scope and sequencing of most courses 
from middle through high school levels is largely dependent upon and driven by the textbook 
that is used (Bean & Zigmond, 1994; Schug, Western, & Enochs, 1997). The difficulty is that 
textbooks do not present information with clear connections among various topics. In the case of 
American history texts, for example, information is typically presented in chronological order 
with one segment of the book only loosely connected to another and without regard to the 
concepts presented in each segment. While chronology seems to be reasonable (though 
potentially ineffective) as an organizing principle in a history text, other content areas have no 
inherent organizational principles. For example, organization in English and science texts as well 
as other areas, is often quite idiosyncratic. Often such materials move from one topic to an 
unrelated topic with little more to organize them than the general content topic (narrative fiction, 
poetry in English or chemistry and then earth science in science texts). 

Organization, then, is the use of structural linkages of topics among units of instruction in 
a single course, across multiple courses within a grade, or across multiple grades. In concept- 
based instruction, organization results directly from the use of complex concepts across multiple 
instances or example sets. By organizing instruction in this way, teachers can provide students 
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with an opportunity to associate instructional outcomes from various units into a cogent whole. 

In our explanatory views, then, with appropriate manipulation of organization to include students 
with disabilities and English language learners, the net effect is likely to be unsustainable. 
Purpose ^ 


The purpose of the survey is clearly identified 


The goal of this study was to develop a survey instrument for use with secondary school 
teachers. This survey is to be used in conjunction with other data gathering methods to measure 
teacher attitude towards working with students, in particular those students with special needs. 
Specifically, the survey is designed to sample from four broad areas that the literature suggests 
play important roles in teacher effectiveness: (a) teachers’ attitude towards aligning their 
curriculum, instruction, and assessment with state content standards ( alignment ), (b) their 
attitude towards working with students having diverse needs ( inclusivity ), (c) their ability to 
explicitly identify the most important concepts in the content they teach ( organization ), and (d) 
their feeling of efficacy in being able to address these three areas (efficacy). 

Methods 

In this section, we first describe the process we used in developing the survey instrument: 
writing items, conducting a content review, and revising the survey. We then describe the 
process we used to test the adequacy of the hypothesized measurement model to describe the 
data we gathered during the piloting of the instru 


Developing the Survey Instrument 


As part of the development process, a draft of 
the survey instrument was reviewed by a 
small number (n=6) of people representative 
of the intended audience for the survey. 


Content Review took place during a two-week window in October. Invitations to review 
the survey were sent to six local teachers representing a variety of content areas and a range of 
experience (See Table 1). All teachers who were invited to review the survey agreed to 
participate. All reviewers were sent a copy of the Feedback Form (See Appendix A) as an email 
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attachment and then provided with a link to the on-line survey itself. Once they had completed 
the review of the survey and responded to the questions on the Feedback Form, teachers emailed 
their reviews to the lead author, who compiled the results and used them to inform revision of the 
survey instrument. 

Table 1 


Demographic Information on Teachers Who Participated in Content Review of Survey 


Reviewer # 

Fevel Taught 

Content Area 

Years Teaching Experience 

Reviewer #1 

middle school 

math 

5 

Reviewer #2 

middle school 

science 

5 

Reviewer #3 

high school 

science 

3 

Reviewer #4 

high school 

language arts 

1 


Once content reviewers had provided their feedback, we revised the survey to reflect their 
suggestions. The final survey that we administered can be found in Appendix C. 

Piloting the Survey Instrument 

We piloted the survey with 104 secondary school teachers in schools across Oregon. 
Teachers were self-selected through a several step process. We sent requests for assistance in 
sending the email invitation to participate in the survey to every secondary school principal with 
a valid email address in the Oregon State Public School Directory. In the request, we asked 
principals to forward the invitation to participate in the survey research to all teachers on their 
staff. We asked teachers to reply directly to the lead researcher by email if they were interested 
in participating in the study. The lead researcher compiled a list of teacher emails for three 
weeks, until she had sufficient numbers of participants to begin the survey. In all, 128 middle 


Steps used to recruit participants 
are described in detail 
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and high school teachers from schools across the state of Oregon indicated their interest in 
participating by emailing the lead researcher. Surveys were initially sent out to all teachers who 
had indicated interest in participating on November 19. Weekly email reminders were sent to all 
participants who had not yet responded, and the final surveys were completed on December 17. 
During the four-week window in which responses were accepted, 105 teachers completed the 
survey (a response rate of 82%). Teachers were paid $10 for their participation in the survey 


research. 


Survey participants are described including response rate 
and timeframe during which survey took place 


Description of our pilot study sample. Survey respondents represent a wide range of 
demographics, Almost twice as many women participated in the survey as men (68 and 35, 
respectively). Tables 2-10 list additional demographic information for the teachers who 
participated in the survey. 


Demographic information of respondents is 
provided 


Table 2 


Self-Reported Ethnicity /Race of Survey Respondents 


Caucasian 

Asian 

Black 

Hispanic 

Chinese/Latina Native 

American 

Decline to 
state 

97 

1 

1 

2 

1 1 

2 


Table 3 


Self-Reported Age of Survey Respondents 


21-30 

31-40 

41-50 

51 + 

Decline to state 

23 

21 

28 

28 

1 
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Table 4 


Self-Reported Highest Level of Education Obtained of Survey Respondents 


BA/BS 

MA/MS/MEd 

DEd/PhD 

22 

81 

1 


Table 5 


Self-Reported Number of Years Teaching , Including Current Year, of Survey Respondents 


0-5 

6-10 

11-20 

21+ 

Decline to state 

33 

25 

28 

15 

2 


Table 6 


Self-Reported Subject Area Taught of Survey Respondents (Grouped by Tested by State 
Assessment, Not Tested by State Assessment, and Special Education ) 


Tested by State Assessment 

Not Tested By State 
Assessment 

Special Education 

72 

26 

5 


Table 7 


Self-Reported Grade Level Taugh t of Survey Responden ts 


Middle School 

High School 

Both Middle and High School 

71 

25 

8 
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Table 8 


Self-Reported Size of School of Survey Respondents 


Fewer than 250 
students 

251 -500 
students 

501 - 1000 
students 

1001 - 1500 
students 

More than 1500 
students 

7 

19 

42 

30 

6 

Table 9 





Self-Reported Type of School of Survey Respondents 



Rural 

Small Town 

Suburban 

Urban 

Decline to state 

24 

34 

41 

4 

1 

Table 10 





Percentage of Students at Survey-Respondents’ Schools Receiving Free/Reduced Priced Meals 

Less than 11% 

- 20% 21% - 30% 

31% -40% 

41% -50% 51% + 

Decline to 

10% 




state 


11 


12 


20 


Evaluating the Stability of the Survey Insi thoughts / experiences 


The level of detail with which we analyzed the survey in 
this study exceeds what is typically required of surveys 
administered in school settings, where the purpose is 
primarily on collecting information about people's 


To evaluate the reliability of the survey instrument, we drew a random sample of 20 
respondents (approximately 20%) from the original participants and sent a duplicate survey 2-4 
weeks after they had completed the survey initially. In all, 16 teachers completed the duplicate 
survey, a response rate of 80%. We calculated the degree of consistency between teachers’ 
responses to the first survey and the duplicate survey using a partial credit- scoring model. 


Because each question on the survey had 4 possible responses, we calculated consistency 
between the initial and duplicate surveys by awarding 100% to each response that was an exact 
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match, 75% to each response that varied by only one response category (such as changing from a 
strongly agree to agree or from a disagree to an agree), 50% to each response that varied by 2 
response categories, and 0% to each response that varied by 3 or more response categories. 
Analyzing the Model Fit 

Our main concern in this study was the creation of a survey instrument to measure 
teachers’ attitudes and perceptions toward a variety of topics that the literature suggested as 
relevant to teacher effectiveness in working with students with diverse needs. To this end, we 
began by running exploratory factor analyses searching for items that appeared to be 
uncorrelated with other items grouped within the four areas we hypothesized would emerge as 
latent factors (alignment, inclusivity, organization, and general efficacy) 1 . For these analyses, we 
used principal component analysis with Quartimax rotation and retained factors with 
Eigenvalues of 1.0 or greater. Once we had determined which items to remove from the survey 
and had a rough idea of the number of factors underlying each construct of interest, we 
constructed a measurement model to test (see Figure 1). Because we had a limited number of 
cases (ranging from 80 on some items to 104 on others), we decided to test each factor 
separately, evaluate model fit, modify the items used in the measurement model as needed, and 
then run the complete model. 

Results 

The next section is organized into two parts. Part 1 presents results evaluating the 
technical adequacy of the survey instrument, in which we evaluate both the reliability and 
adequacy of the survey instrument and the degree to which the hypothesized measurement model 


1 We included questions about state testing on the survey to answer other research questions, but 
they are not part of the measurement model 
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fit the data. Part 2 presents teachers’ responses to the items on the survey during the 
administration of the pilot test. 

Part 1: Technical Adequacy 

Content review and revision of the survey. The compiled results from the content review 
process are presented below. Where appropriate we present feedback from reviewers in tabular 

form (see Tables 1 1 - 15). The rest of the time, responses are presented in full in the reviewers’ 

Here, we elected to provide the 
constructed responses in full, to 
provide as complete a picture as 
possible about the feedback received 
during the survey creation process 

• There needs to be some kind of introduction as to what the survey is about. Also, the 
e-mail that I got with the link was not titled “U of O Research,” it was titled “Survey 
from Infocounts.” [Reviewer #1] 

• Maybe a short introduction explaining what the survey is about (people may have 
forgotten that they signed up to take this survey). [Reviewer #2] 

• Take off this part: “InfoCounts is now available for free to the general public! This 
simple, easy to use, online survey tool can collect data from anyone with an email 
address. Use the free InfoCounts system to increase productivity, conserve resources, 
and find out what really counts! http://www.infocounts.com”. This seems to add 
more confusion than help. [Reviewer #3] 

• The email seems very clear. I’m not sure if you want to say that the answers will 
remain anonymous. I may be out to lunch on this one, but can answers remain 
anonymous? Would confidential be more accurate? [Reviewer #4] 


exact wording. 

Feedback on the survey instrument. 

1. What changes, if any, would you recommend be 


Teacher Efficacy Survey 15 


Table 11 


Reviewers’ Feedback on Question: “How easy is it to understand what you are expected to do?” 


Reviewer # not at all easy 

somewhat easy 

easy very easy 

1 


X 

2 


X 

3 

X 


4 


X 

Survey Questions 1-9 



Table 12 

Reviewers’ Feedback on Question: 

“How easy to understand were the directions?” 

Reviewer # not at all easy 

somewhat easy 

easy very easy 

1 


X 

2 

X 


3 


X 

4 


X 


2. What changes, if any, would you recommend be made to the directions at the start of this 
section? 

• None [Reviewer #1, #2, #3] 


3. Which questions, if any, did you find confusing? Why? 

• Question 6, says only text value, but an answer requires a numeric part as well as a 
text part. Again on Question 9, it asks for only text value, but the answer would be a 


numeric answer. [Reviewer#!] 
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• Question #6 and #9 ask for a number, but the directions say to enter “only text value.” 
I may misunderstand, but it seems like you are asking for a number but want it 
written out (i.e. you want the number “8”, but you want the person to respond with 
“eight.” This is confusing. [Reviewer #2] 

• None [Reviewer #3] 

• Just the possible semantic tweaking with “anonymous” [Reviewer #4] 

4. On which questions, if any, would you recommend changing the wording? 

• See answer to #3 [Reviewer #2] 

• None [Reviewer #3, #4] 

a. How should the wording be changed? 
o See answer to #3 [Reviewer #2] 

5. Are there any other demographic questions that you think we should add to this section? 

• I don’t know how you are planning to use this information, but some other areas to 
consider would be family structure, sexual orientation, and income level. [Reviewer 
# 2 ] 

• None [Reviewer #3, #4] 

6. Approximately how long would it take you to complete this section of the survey itself? 

• A couple of minutes [Reviewer #1] 

• One minute [Reviewer #2] 

• A couple minutes [Reviewer #3, #4] 
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Survey, Question 10 
Table 13 


Reviewers’ Feedback on Question: “ How easy to understand were the directions in Question 
10 ?” 


Reviewer # 

not at all easy 

somewhat easy 

easy 

very easy 

1 




X 

2 



X 


3 

X 




4 



X 



7. What changes, if any, would you recommend to the wording of Question 10? 

• This almost seems unnecessary to include as a question, maybe just a button that says 
“next.” [Reviewer #2] 

• This question totally confused me because nothing happened when I 
checked/unchecked the box. I first checked it thinking it was going to go to another 
page or something, but then nothing happened. It was confusing to see that I needed 
to check “next” when all the questions I was going to be answering were right there 
on the same page. [Reviewer #3] 

• You might explain the purpose of the question. [Reviewer #4] 

Survey, Questions 11-20 

8. Which questions, if any, did you find confusing? Why? 

• #1 1 - does “well prepared” mean that you have the formal training to teach it, that 
you have the experience to teach it, that there are resources at the school that help 
prepare you to teach it, etc? Maybe this could be specified [Reviewer #2] 
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• #20 - this is actually asking two different questions. Maybe break it up into two 
questions or get rid of one of the statements. [Reviewer #2] 

• #13. Not confusing, but bothered me. I understood what the question was getting at, 
but it was hard for me to answer honestly. I do feel like I “try really hard,” but 
sometimes there are some students that just can not be reached for whatever reason. 
Answering this question with the options given was difficult. I felt like I needed to be 
able to write in text to support my answer. Maybe a text box could be added or the 
wording of the question could be changed. [Reviewer #3] 

• This section seems to be effectively worded. [Reviewer #4] 

9. On which questions, if any, would you recommend changing the wording? 

• See answer to #8 [Reviewer #2] 

• #13 [Reviewer #3] 

• None [Reviewer #4] 

a. How should the wording be changed? 
o See answer to #8 [Reviewer #2] 

o Changing the part about “If I try really hard.” Maybe clarifying what “trying 
really hard” means by adding something about “by using various teaching 
strategies all of my educational knowledge” or “to the best of my ability.” 
[Reviewer #3] 

10. Which questions, if any, did you find redundant in this section? 

• None [Reviewer #1, #2, #3, #4] 

1 1 . Approximately how long would it take you to complete this section of the survey itself? 

• A couple of minutes. [Reviewer #1] 


One minute [Reviewer #2] 
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• Couple minutes [Reviewer #3] 

• Approximately 4 minutes [Reviewer #4] 

Survey, Questions 21-30 

12. Which questions, if any, did you find confusing? Why? 

• #22 - “water down” might not be a universally known phrase. . .maybe use something 
like “simplify” or “not make as complex.” [Reviewer #2] 

• None [Reviewer #3] 

• I found this section to be extremely reassuring. It made me feel like I planned my 
curriculum well. [Reviewer #4] 

13. On which questions, if any, would you recommend changing the wording? 

• See answer to #12 [Reviewer #2] 

a. How should the wording be changed? 
o See answer to #12 [Reviewer #2] 

o You might consider asking about specific challenges i.e. autism, dyslexia, etc. 
[Reviewer #4] 

14. Which questions, if any, did you find redundant in this section? 

• None [Reviewer #1, #2, #3, #4] 

15. Approximately how long would it take you to complete this section of the survey itself? 

• A couple of minutes. [Reviewer #1] 

• Two minutes [Reviewer #2] 

• Couple minutes [Reviewer #3] 

• Three minutes [Reviewer #4] 

Survey, Questions 31-40 

16. Which questions, if any, did you find confusing? Why? 
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• On question 38, there was only one option for an answer, which was strongly 
disagree. Also, question 39 is exactly the same as 38. [Reviewer #1] 

• Question #38 only has one choice for an answer: strongly disagree. . .change by giving 
all four available responses just like the other questions [Reviewer #2] 

• Question #38 only gives you one option to choose from. [Reviewer #3] 

• Question thirty two is a bit vague in its reference to “text”. What if you do not have a 
traditional textbook, [reviewer #4] 

17. On which questions, if any, would you recommend changing the wording? 

• See answer to #16 [Reviewer #2] 

• #32 [Reviewer #4} 

a. How should the wording be changed? 
o See answer to #16 [Reviewer #2] 
o Throw #38 out? [Reviewer #3] 

18. Which questions, if any, did you find redundant in this section? 

• Question 33 seems to be asking the same question as 38. The same question seems to 
be asked again in question 40. [Reviewer #1] 

• #32 and #35 and #40 are all asking different questions, but they seem redundant. 
Maybe by placing them one after another it would be easier to see the differences. 
Also, #38 and #39 are the same questions [Reviewer #2] 

• #38 and #39 [Reviewer #3, #4] 

19. Approximately how long would it take you to complete this section of the survey itself? 

• 4 or 5 minutes [Reviewer #1] 

• Two minutes [Reviewer #2] 


Couple of minutes [Reviewer #3] 
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• Two minutes [Reviewer #4] 

Survey, Questions 41-50 

20. Which questions, if any, did you find confusing? Why? 

• #48 - it is posed as a DOES NOT question, so disagreeing with it is 
confusing. . .maybe make it a DOES question. [Reviewer #2] 

• None [Reviewer #3] 

21. On which questions, if any, would you recommend changing the wording? 

• See answer to #20 [Reviewer #2] 

• #49 [Reviewer #3] 

• Teachers may not have all of the information to answer question 42. How might they 
communicate this? [Reviewer #4] 

a. How should the wording be changed? 
o See answer to #20 [Reviewer #2] 

o Clarify option “improved materials selection”. Is this referring to more/better 
texts, etc.? [Reviewer #3] 

22. Which questions, if any, did you find redundant in this section? 

• None [Reviewer #2, #3] 

• Questions 41 and 44 may be redundant. [Reviewer #4] 

23. Approximately how long would it take you to complete this section of the survey itself? 

• Two minutes [Reviewer #2, #4] 


Couple minutes [Reviewer #3] 
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Overall Feedback 
Table 14 


Reviewers’ Feedback on Question: “ How easy was it to use the on-line survey?” 


Reviewer # not at all easy 

somewhat easy 

easy 

very easy 

1 



X 

2 

X 



3 

X 



4 


X 


Table 15 




Reviewers’ Feedback on Question: 
school teachers?” 

“ How appropriate is the survey to use 

with middle and high 

Reviewer # not at all 

appropriate 

somewhat 

appropriate 

appropriate 

very appropriate 

1 



X 

2 


X 


3 


X 


4 


X 



24. What changes in formatting/ appearance, etc. would improve the on-line survey? 

• I thought the appearance and formatting was easy to read, and user-friendly. 
[Reviewer #1] 

• Maybe make a space between questions and a space between the question and its 


possible answers [Reviewer #2] 
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• The end piece when you click “reply.” There is nothing instructing you to do so, and 
when you do (click reply) you are left unsure if you did the right thing. It seems that 
you are done, but are not quite sure about it. Maybe a screen that says “Thank you 
for completing the survey. Your survey is now complete and accounted for. You 
may exit this screen.” Something like that would be helpful. [Reviewer #3] 

25. What changes in the directions would improve the on-line survey? 

• None different that the suggestions already given [Reviewer #2] 

• Perhaps you should say “If you do not have the information or experience to answer a 
question, leave it blank.” [Reviewer #4] 

26. Please note any additional suggestions you have for us. Thanks! 

• Thanks for your patience in my responding! I did eventually figure out what you 
were asking me to do! I think with some changes to better clarify the survey, it will 
be a keeper! [Reviewer #3] 

Provide as much direct, contextual information as possible in the survey. For instance, it might 
be worth it to clarify that the results may be used to existing programs at the University of 
Oregon School of Ed. [Reviewer #4] 

Stability of the survey instrument. In all, 16 teachers completed the duplicate survey, a 
response rate of 80%. We calculated the degree of consistency between teachers’ responses to 
the first survey and their responses to the duplicate survey using a partial credit scoring model. 
The majority of responses on the two survey administrations were exact matched, but three 
teachers’ responses varied by one response category in about one quarter of the possible 53 
responses. In all, the survey was found to have a test/re-test reliability of 91% for the 16 teachers 
who completed both the original and duplicate survey. 
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Testing of a Measurement Model 

In this section, we first present the results of our initial principal component analyses and 
then present the results of our path analyses as we move into evaluation of the full measurement 
model 

Results of exploratory factor analysis. Only 6 items (of the n items) were identified as 
not functioning well in our survey: one each in the efficacy (item 11) and inclusivity (item 15) 
sections of the survey, and one in the alignment (items 25 and 28) section. In addition, two items 
from the questions on state standards / state testing (items 32 and 33) were dropped because they 
did not appear to be functioning well. 

Principal component analysis with Quartimax rotation resulted in the identification of 
three underlying factors that together comprised the ejficacy construct (see Table 16), one 
underlying factor that comprised the inclusivity construct (see Table 17), two underlying factors 
that together comprised the organization construct (see Table 18), one underlying factor that 
comprised the alignment construct (see Table 19), and three underlying factors that together 
comprised the state standards/ state testing construct (see Table 20). All analyses converged in 
four iterations. In all cases, we only retained factors with initial Eigenvalues of at least 1.00. 
Eigenvalues for the individual factors are indicated in Tables 16 - 20. 
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Table 16 


Principal Component Analysis, Efficacy 


Survey Item 

Component (Initial Eigenvalues) 

1 (2.76) 

2(1.63) 

3 (1.09) 

Overall, I am comfortable teaching the curriculum in my 
assigned classes. (v3) 

.93 

.16 

-8.84E-03 

I am well prepared to teach my assigned content classes. 
(v4) 

.91 

.16 

-1.08E-02 

The amount a student can learn is primarily related to 
family background. (v5) (reverse scored) 

.19 

.61 

.15 

If I try really hard I can get through to even the most 
difficult or unmotivated students. (v6) 

1.92E-02 

.79 

.16 

If a student in my class becomes disruptive and noisy I 
know effective techniques to redirect him/her quickly. (v7) 

Poorly fitting item. Cut from 
Survey. 

When one of my students gets a better grade than he/she 
typically gets it is usually because I found a better way of 
teaching that student. (v8) 

.16 

.71 

-8.72E-02 

If a student did not remember information I gave in a 
previous lesson I would know how to increase his/her 
retention in the next lesson. (v9) 

.16 

-3.01E-02 

.81 

When a student is having difficulty with an assignment I am 
able to adjust it to his/her level of skill. (vlO) 

-.11 

.26 

.73 

When it conies right down to it a student’s motivation and 
performance depends on his/her home environment not on 
the teacher, (vll) (reverse scored) 

Poorly fitting item. Cut from 
Survey. 
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Table 17 


Principal Component Analysis, Inclusivity 


Survey Item 

Component (Initial 
Eigenvalue) 


1 (1.98) 

Some students in my classes would be better placed with a teacher 
who has additional training in meeting the needs of students with 
learning disabilities. (vl2) (reverse scored) 

.50 

I need additional support (e.g. classroom aides) to address the needs 
of all students in my classes. (vl3) (reverse scored) 

.70 

Having students with special needs in my classroom has negative 
consequences for other students. (vl4) (reverse scored) 

.81 

All my students can learn the main ideas in my curriculum. (vl5) 

Poorly fitted item. Cut 
from survey. 

For students with special needs to succeed I have to take instructional 
time away from other students. (vl6) (reverse scored) 

.76 
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Table 18 


Principal Component Analysis, Organization 


Survey Item 

Component (Initial 
Eigenvalues) 


1 (2.76) 

2(1.34) 

The main ideas of the curriculum in my content area(s) are addressed 
across the units in my class. (vl7) 

.71 

-4.67E-.03 

I review the big picture of my curriculum to ensure that there is 
coherence across lessons in my class. (vl8) 

.82 

.19 

I teach my students to look for the same main ideas across multiple 
lessons. (vl9) 

.68 

7.06E-02 

My lessons are designed to explicitly support the broader goals within 
my curriculum. (v20) 

.77 

.28 

I make sure that my tests measure the main ideas that I want students 
to master. (v22) 

5.41E-02 

.67 

I use specific techniques (e.g. outlines graphic organizers note taking) 
to make the main ideas obvious in my class. (v23) 

.17 

.73 

I provide my students with multiple opportunities to practice using the 
main ideas of the curriculum in my content area. (v24) 

.11 

.77 
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Table 19 


Principal Component Analysis, Alignment 



Component 


(Initial 

Survey Item 

Eigenvalues) 


1 (4.28) 

I modify the curriculum in my classes to match the state standards. (v21) 

.80 

I primarily use a textbook to identify what it important to teach in my classes. 
(v25) (reverse scored). 

Poorly fitted 
item. Cut from 

survey. 

I primarily use the state standards to identify what is important to teach in my 
classes. (v26) 

.87 

My curriculum, instruction, and assessments are aligned to the main ideas in 
the state standards. (v27) 

.72 

I am quite familiar with the main ideas in the state standards for my assigned 
content area(s). (v28) 

Poorly fitted 
item. Cut from 

survey. 

I develop lesson plans to teach students the main ideas in the state standards. 
(v29) 

.90 

I plan assessments to measure student mastery of main ideas relative to the 
state standards. (v30) 

.87 

I use the state standards to identify what is important to assess in my classes. 

(v31) 

.90 
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Table 20 


Principal Component Analysis, State Standards / State Testing 


Survey Item 

Component (Initial Eigenvalues) 

1 (2.04) 

2(1.53) 

3 (1.26) 

For students in my classes, improvement on state tests is 
extremely difficult to accomplish. (v36) (reverse scored). 

.89 

2.74E-02 

-3.55E-02 

The state’s expectations are unreasonable for students at my 
school (reverse scored). (v37)) 

.84 

1.34E-02 

.18 

I know how students at my school compare to students at 
other schools in my state on state tests. (v38) 

.12 

.85 

9.35E-.02 

I know how students at my school compare to students 
across the nation in terms of their performance on 
standardized tests. (39) 

-7.51E-02 

.87 

-.10 

State testing appropriately measures my students’ strengths 
and weaknesses. (v40) 

.15 

.11 

.80 

The statewide assessment addresses the main ideas of state 
content standards. (v41) 

-1.16E-02 

-.12 

.86 

The state’s expectations are reasonable for students in my 
assigned content area. (v32) 

Poorly fitting item. Cut from 
survey. 

I think students at my school are doing as well as ever 
academically. (v33) 

Poorly fitting item. Cut from 
survey. 


Results of measurement model analyses. For the alignment construct, we achieved the 
best model fit (x 2 = 19.24, p < .05; CFI = .97; RMSEA = .12) when we let the residual from v26 
covary with the residual from v29. For the inclusivity construct, we achieved the best model fit 
(X 2 = 2.56, p > .05; CFI = .99; RMSEA = .05) when we used all four measured variables. For the 
organization construct, we achieved the best model fit (x 2 = 4.20, p > .05; CFI = 1.0; RMSEA < 
.001) when we let two sets of residuals covary (vl7 with vl8 and vl9 with v20), and dropped 
one measured variable (v22) from the second factor.For the efficacy construct, we achieved a 
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good model fit (x 2 = 8.85, p > .05; CFI = .97; RMSEA = .11) when we used just five measured 
variables (v3 - v6 and v8), dropping the two measured variables that loaded on the third factor 
(v9, vlO) and one measured variable that didn’t have a single strong factor loading. Although we 
gathered data on teachers’ opinions of state testing, we did not include them in analyses because 
they were not part of our measurement model. 

When we combined all latent constructs into one model, the fit indices went down 
considerably (x 2 = 238.07,/) < .05; CFI = .91; RMSEA = .07), as would be expected given the 
relatively small sample size being used to estimate a rather complicated model. Figure 2 shows 
the empirical path diagram illustrating the relationship between measured variables in our model. 
We have included the output files for all relevant Mplus analyses as Appendix D. 
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Figure 1. Hypothesized Measurement Model 
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Figure 2. 

Empirical Path Diagram 
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Results on Survey During Pilot Testing: How Teachers Responded 


Teachers in the sample provided a range of responses on almost all items on the survey 


(see Table 21). 
Table 21 


Here, results are reported from the survey administration, with 
both raw numbers and percentages of respondents per 
response category, as suggested. Note, too, the use of four 
possible response options for these Likert-scale type items 


Teachers’ Responses to Survey Questions, Numbers and Percentages Per Response Category 


Survey Item 

Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

Overall, I am comfortable teaching the curriculum 
in my assigned classes. (v3) 

6 (6%) 

2 (2%) 

29 

(28%) 

67 

(64%) 

I am well prepared to teach my assigned content 
classes. (v4) 

6 (6%) 

4 (4%) 

35 

(34%) 

58 

(56%) 

The amount a student can learn is primarily related 
to family background. (v5) 

4 (4%) 

21 (20%) 

63 

(61%) 

16 

(15%) 


< 25% of 
the time 

26 - 50% 

51 - 

>75% 


of the 

75% of 

of the 


time 

the time 

time 

If I try really hard I can get through to even the 
most difficult or unmotivated students. (v6) 

12 (12%) 

32 (31%) 

34 

(33%) 

26 

(25%) 


Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

If a student in my class becomes disruptive and 
noisy I know effective techniques to redirect 
him/her quickly. (v7) 

1 (1%) 

10(10%) 

55 

(53%) 

38 

(37%) 

When one of my students gets a better grade than 
he/she typically gets it is usually because I found a 
better way of teaching that student. (v8) 

3 (3%) 

37 (36%) 

59 

(57%) 

3 (3%) 

If a student did not remember information I gave 
in a previous lesson I would know how to increase 
his/her retention in the next lesson. (v9) 

— 

18 (17%) 

78 

(75%) 

7 (7%) 
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Table 21, Cont. 


Teachers’ Responses to Survey Questions, Numbers and Percentages Per Response Category 



< 25% of 
the time 

26 - 50% 
of the 
time 

51 - 
75% of 
the time 

>75% 
of the 
time 

When a student is having difficulty with an 
assignment I am able to adjust it to his/her level of 
skill. (vlO) 

5 (5%) 

21 (20%) 

36 

(35%) 

42 

(40%) 


Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

When it conies right down to it a student’s 
motivation and performance depends on his/her 
home environment not on the teacher, (vl 1) 

6 (6%) 

62 (60%) 

29 

(28%) 

4 (4%) 

Some students in my classes would be better 
placed with a teacher who has additional training 
in meeting the needs of students with learning 
disabilities. (vl2) 

11 (11%) 

36 (35%) 

41 

(39%) 

14 

(14%) 

I need additional support (e.g. classroom aides) to 
address the needs of all students in my classes. 
(vl3) 

19(18%) 

53 (51%) 

26 

(25%) 

5 (5%) 

Having students with special needs in my 
classroom has negative consequences for other 
students. (vl4) 

3 (3%) 

27 (26%) 

55 

(53%) 

19 

(18%) 

All my students can learn the main ideas in my 
curriculum. (vl5) 

2 (2%) 

7 (7%) 

58 

(56%) 

35 

(34%) 

For students with special needs to succeed I have 
to take instructional time away from other 
students. (vl6) 

9 (9%) 

57 (55%) 

30 

(29%) 

7 (7%) 
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Teachers’ Responses to Survey Questions, Numbers and Percentages Per Response Category 



< 25% of 
the time 

26 - 50% 

51 - 

>75% 


of the 

75% of 

of the 


time 

the time 

time 

The main ideas of the curriculum in my content 
area(s) are addressed across the units in my class. 
(vl7) 

2 (2%) 

11 (11%) 

31 

(30%) 

58 

(56%) 

I review the big picture of my curriculum to ensure 
that there is coherence across lessons in my class. 
(vlB) 

8 (8%) 

13 (13%) 

33 

(32%) 

50 

(48%) 


Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

I teach my students to look for the same main 
ideas across multiple lessons. (vl9) 

— 

12 (12%) 

58 

(56%) 

29 

(28%) 


< 25% of 
the time 

26 - 50% 

51 - 

>75% 


of the 

75% of 

of the 


time 

the time 

time 

My lessons are designed to explicitly support the 
broader goals within my curriculum. (v20) 

1 (1%) 

14 (14%) 

35 

(34%) 

52 

(50%) 

I modify the curriculum in my classes to match the 
state standards. (v21) 

16 (15%) 

14 (14%) 

36 

(35%) 

32 

(31%) 

I make sure that my tests measure the main ideas 



14 

85 

that I want students to master. (v22) 



(14%) 

(82%) 

I use specific techniques (e.g. outlines graphic 
organizers note taking) to make the main ideas 
obvious in my class. (v23) 

3 (3%) 

12 (12%) 

28 

(27%) 

56 

(54%) 

I provide my students with multiple opportunities 
to practice using the main ideas of the curriculum 
in my content area. (v24) 

3 (3%) 

3 (3%) 

30 

(29%) 

66 

(64%) 

I primarily use a textbook to identify what it 
important to teach in my classes. (v25) 

9 (9%) 

17 (16%) 

27 

(26%) 

48 

(46%) 
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Table 21, Cont. 





Teachers’ Responses to Survey Questions, Numbers and Percentages Per Response Category 


< 25% of 
the time 

26 - 50% 

51 - 

>75% 


of the 

75% of 

of the 


time 

the time 

time 

I primarily use the state standards to identify what 
is important to teach in my classes. (v26) 

21 (20%) 

18 (17%) 

33 

(32%) 

29 

(28%) 

My curriculum, instruction, and assessments are 
aligned to the main ideas in the state standards. 
(v27) 

3 (3%) 

10 (10%) 

39 

(38%) 

47 

(45%) 


Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

I am quite familiar with the main ideas in the state 
standards for my assigned content area(s). (v28) 

2 (2%) 

6 (6%) 

46 

(44%) 

47 

(45%) 


< 25% of 
the time 

26 - 50% 

51 - 

>75% 


of the 

75% of 

of the 


time 

the time 

time 

I develop lesson plans to teach students the main 
ideas in the state standards. (v29) 

11 (11%) 

16 (15%) 

36 

(35%) 

37 

(36%) 

I plan assessments to measure student mastery of 
main ideas relative to the state standards. (v30) 

11 (11%) 

28 (27%) 

34 

(33%) 

28 

(27%) 

I use the state standards to identify what is 
important to assess in my classes. (v31) 

18 (17%) 

21 (20%) 

36 

(35%) 

25 

(24%) 


Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

The state’s expectations are reasonable for students 
in my assigned content area. (v32) 

6 (6%) 

18 (17%) 

62 

(60%) 

14 

(14%) 

I think students at my school are doing as well as 
ever academically. (v33) 

3 (3%) 

30 (29%) 

55 

(53%) 

12 

(12%) 

For students in my classes, improvement on state 
tests is extremely difficult to accomplish. (v36) 

5 (5%) 

32 (31%) 

53 

(51%) 

5 (5%) 
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Table 21, Cont. 


Teachers’ Responses to Survey Questions, Numbers and Percentages Per Response Category 



Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

The state’s expectations are unreasonable for 
students at my school (reverse scored). (v37)) 

5 (5%) 

22 (21%) 

60 

(58%) 

8 (8%) 

I know how students at my school compare to 
students at other schools in my state on state tests. 
(v38) 

6 (6%) 

10 (10%) 

70 

(67%) 

16 

(15%) 

I know how students at my school compare to 
students across the nation in terms of their 
performance on standardized tests. (39) 

12 (12%) 

41 (39%) 

41 

(39%) 

6 (6%) 

State testing appropriately measures my students’ 
strengths and weaknesses. (v40) 

26 (25%) 

55 (53%) 

21 

(20%) 

— 

The statewide assessment addresses the main ideas 
of state content standards. (v41) 

3 (3%) 

24 (23%) 

61 

(59%) 

1 (1%) 


Teachers’ Estimation of The Likelihood of Their Students Failing the State Test 

We also asked teachers to estimate the percentage of their students whom they believe 
would fail the state large-scale assessment in spite of everything they did as teachers (see Table 
22). Responses ranged from a low of 0 to a high of 75% (M=28.1 1, SD = 17.98). 
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Table 22 


Percentage of Students Whom Teachers Believe Will Fail the State Test, Irrespective of 
Teachers’ Efforts 


Percentage of Students Who 
Will Fail 

Number of % of Teachers with 

Teachers with This This Response 

Response 

Cumulative % 

<5 

7 

8.2 

8 

Between 5 and 10 

14 

16.5 

24.7 

Between 11% and 15% 

10 

11.8 

36.5 

Between 16% and 20% 

5 

5.9 

42.4 

Between 21% and 25% 

7 

8.2 

50.6 

Between 26% and 30% 

11 

12.9 

63.5 

Between 31% and 35% 

5 

5.9 

69.4 

Between 36% and 40% 

6 

7.1 

75.5 

Between 41% and 45% 

3 

3.5 

79.0 

Between 46% and 50% 

10 

11.8 

90.8 

Between 51% and 55% 

0 

0 

90.8 

Between 56% and 60% 

5 

5.9 

96.7 

Between 61% and 65% 

1 

1.2 

97.9 

Between 66% and 70% 

0 

0 

97.9 

Between 71% and 75% 
>75% 

1 

0 r 

3 

C\C\ 1 a 

Here, you see an example of analysis of 
patterns of responses by specific 

rlpmnnranhir. r.harar.tpriptir.s nf rppnnnrlpntp 

“Less than 100% due to rounding 

errors. / 




Interestingly, responses to this question varied by type of district in which the teachers 
worked, with teachers working in rural (n = 19) and small town (n = 30) districts on average 
expressing the least optimism (M= 37.37, SD = 15.93 andM= 31.67, SD = 20.32, respectively, 
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p < .05). On average, teachers working in suburban ( n = 33) districts estimated the smallest 
percentage of students would fail the state test in spite of everything they do (M= 19.82, SD = 
12.93 ,p < .05). We found no statistically significant differences in teachers’ estimation of the 
percentage of their students who would fail the state test regardless of teacher efforts with 
regards to years of teaching or size of school (p > .05). 

Perhaps not surprisingly, however, given the research indicating a consistent relationship 
between SES and achievement on standardized tests, there was a trend in our sample for teachers 
who worked at schools with greater proportion of students receiving free or reduced price meals 
to indicate less optimism about the percentage of their students who would pass the state test (p < 
.05). On average, teachers who reported they worked at schools with fewer than 10% of their 
students receiving free or reduced price meals estimated that 15% of their students would fail the 
state test. In contrast, teachers who reported working at schools with more than 50% of their 
students receiving subsidized meals estimated that 35% of their students would fail the state test. 
Full results are presented in Table 23 
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Table 23 


Teachers’ Estimations of the Percentage of Their Students Who Will Fail the State Test in Spite 
of Everything [They] Do 


Approximate percentage of students receiving free or reduced 
price lunch 

n 

M 

SD 

less than 10% 

8 

15.25 

15.65 

1 1 - 20% 

8 

20.25 

16.79 

21-30% 

19 

22.58 

15.17 

31-40% 

12 

35.58 

20.75 

41-50% 

9 

33.89 

20.73 

more than 50% 

24 

35.13 

15.83 


Discussion 

Results from our pilot testing of the survey instrument suggest the revised survey (see 
Appendix E) may be a technically adequate measure of secondary school teachers’ perceptions 
of their efficacy working with students from diverse backgrounds. The survey provides reliable 
information about teachers’ self-reported ability to align curriculum, instruction, and assessment; 
their feeling of inclusivity; their self-reported ability to organize their instructional units to 
reflect state content standards; and their general efficacy. However, the small size of our sample 
limits our ability to test the fit of our model in its entirety. The fit indices nearing significance 
indicate a general trend toward model fit, but further empirical testing with a larger sample will 
be necessary to fully test the complete model. 
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Appendix A: Feedback Form Used During Content Review 

Feedback on CITES Teacher Survey 

Thank you for your assistance in our efforts. The on-line survey about which you will be 
providing feedback is designed for in-service secondary school teachers from a variety of 
content areas. Your feedback will help us in the editing/ revising phase of our work, as we 
prepare for a pilot testing of this instrument beginning in mid October. The survey is being 

developed as part of a grant-funded research project being conducted by of in the 

College of Education at . 

Instructions: 

1 . Please read over the questions on this feedback form prior to looking at the survey to 
familiarize yourself with the types of questions you will be answering. You will see that 
we ask the same questions for each of the sections of the survey, focusing on the 
formatting and wording of the items. 

2. Once you know what to expect, please move through the survey instrument, providing 
feedback on each section and for each item. You may either write your responses on the 
printed copy of this feedback form or type your responses on the electronic version of the 
form that has been provided in a separate email. 

3. You will access the survey on your computer. Please note: although you will need to 
provide answers to the survey questions in order for the computer program to function 
appropriately, your answers on the survey itself will NOT be used in any way. For this 
reason and because we want you to devote your time to providing feedback rather than 
contemplating how you should answer the survey, please feel free to provide fictitious 
answers on the survey itself. 

4. As soon as you have completed the feedback form, please return your responses to me, 

, at . For faster delivery, responses can also be sent to me at as email 

attachments (with a .doc extension). 

If you have any questions, please contact me at email or phone number. 

Thank you for providing us with feedback! 

Sincerely, 


Name, Title 
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Reviewer Information 

Name: Date: 

Current job title: # of years at current job: 

If you are a teacher, what subject do you teach? 

Feedback on the Survey Instrument 

E-mail Intro 

Please read over the e-mail with the subject heading “U of O Research” that contains the link to 
the on-line survey. This e-mail will be teachers’ first introduction to the survey. With this context 
in mind: 

1. What changes, if any, would you recommend be made to the e-mail? 


2. How easy is it to understand what you are expected to do? 

not at all easy somewhat easy easy very easy 

Survey Questions 1 - 9 

Please read over questions 1-9 of the survey. These questions are designed to gather 
demographic information . 

3. How easy to understand were the directions? 

not at all easy somewhat easy easy very easy 

4. What changes, if any, would you recommend be made to the directions at the start of this 
section? 

5. Which questions, if any, did you find confusing? Why? 

6. On which questions, if any, would you recommend changing the wording? 

a. How should the wording be changed? 

7. Are there any other demographic questions that you think we should add to this section? 
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8. Approximately how long would it take you to complete this section of the survey itself? 

Survey, Question 10 

Question 10 on the Survey is used to overcome a technical difficulty with the on-line 
format— including this question allows me to provide a ‘break’ in the survey between the 
demographics and the actual survey questions as well as to give directions for how to proceed 
with the rest of the survey. Please read Question 10 and then respond to the following questions: 

9. How easy to understand were the directions in Question 10? 

not at all easy somewhat easy easy very easy 


10. What changes, if any, would you recommend to the wording of Question 10? 


Survey, Questions 11-20 

11. Which questions, if any, did you find confusing? Why? 


12. On which questions, if any, would you recommend changing the wording? 


b. How should the wording be changed? 


13. Which questions, if any, did you find redundant in this section? 


14. Approximately how long would it take you to complete this section of the survey itself? 


Survey, Questions 21-30 

15. Which questions, if any, did you find confusing? Why? 
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16. On which questions, if any, would you recommend changing the wording? 


c. How should the wording be changed? 


17. Which questions, if any, did you find redundant in this section? 


18. Approximately how long would it take you to complete this section of the survey itself? 


Survey, Questions 31 - 40 

19. Which questions, if any, did you find confusing? Why? 


20. On which questions, if any, would you recommend changing the wording? 


d. How should the wording be changed? 


21. Which questions, if any, did you find redundant in this section? 


22. Approximately how long would it take you to complete this section of the survey itself? 


Survey, Questions 41 - 50 

23. Which questions, if any, did you find confusing? Why? 


24. On which questions, if any, would you recommend changing the wording? 
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e. How should the wording be changed? 


25. Which questions, if any, did you find redundant in this section? 

26. Approximately how long would it take you to complete this section of the survey itself? 

Overall Feedback 

Please consider the on-line format of the survey in addressing the following questions: 

27. How easy was it to use the on-line survey? 

not at all easy somewhat easy easy very easy 

28. How appropriate is the survey to use with middle and high school teachers? 

not at all somewhat appropriate very appropriate 

appropriate appropriate 

29. What changes in formatting/ appearance, etc. would improve the on-line survey? 

30. What changes in the directions would improve the on-line survey? 

31. Please note any additional suggestions you have for us. Thanks! 


As soon as you have completed the feedback form, please return your responses to at 

. For faster delivery, responses can also be sent to me at as email attachments 

(with a .doc extension). 


Thank you! 
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Appendix B: Technical Specifications of Survey 

The following questions were included to gather demographic information: [number of item on 

actual survey ], ( variable identification on Empirical Path Diagram shown in Figure 2 ) 

Explanation of the 
intended purpose 
of each item on the 
survey 

4. Highest degree earned [48] 

5. Teaching licensure (grade levels/disciplines) [50] 

6. Years teaching [49] 

7. Currently assigned content area(s) [51] 

8. Grade level(s) of students you are teaching this year [52] 

9. Approximately how many students attend your school? [53] 

10. Approximately what percentage of students at your school receive free/reduced price lunch? 
[54] 

11. How would you characterize your school’s setting [rural, small town, suburban, urban] [55] 

The following questions were included to gather information about efficacy: 

12. 1 am well prepared to teach my assigned content classes [4] (v4) 

13. Overall, I am comfortable teaching the curriculum in my assigned classes [3] (v3). 

14. If I try really hard, I can get through to even the most difficult or unmotivated students [6] 
(v6) 

15. If a student in my class becomes disruptive and noisy, I know effective techniques to redirect 
him/her quickly. [7] (v7) 

16. When one of my students gets a better grade than he/she typically gets, it is usually because I 
found a better way of teaching that student. [8] (v8) 

17. If a student did not remember information I gave in a previous lesson, I would know how to 
increase his/her retention in the next lesson. [9] (v9) 

18. When a student is having difficulty with an assignment, I am able to adjust it to his/her level. 
[10] (vlO) 

19. When it comes right down to it, a student’s motivation and performance depends on his/her 
home environment not on the teacher. [11] (vl 1) 


1. uenaer ptoj 

2. Age [46] 

3. Ethnicity [47] 
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20. The amount a student can learn is primarily related to family background. [5] (v5) 

The following questions were included to gather information about inclusivity: 

21. Some students in my classes would be better placed with a teacher who has additional 
training in meeting the needs of students with learning disabilities. [12] (vl2) 

22. 1 need additional support (e.g. classroom aides) to address the needs of all students in my 
classes. [13] (vl3) 

23. All my students can learn the main ideas in my curriculum. [15] (vl5) 

24. For students with special needs to succeed, I have to take instructional time away from other 
students. [16] (vl6) 

25. Having students with special needs in my classroom has negative consequences for other 
students. [14] (vl4) 

The following questions were included to gather information about organization: 

26. The main ideas of the curriculum in my content area(s) are addressed across the units in my 
class: [17] (vl7) 

27. 1 review the big picture of my curriculum to ensure that there is coherence across lessons in 
my class: [18] (vl8) 

28. 1 teach my students to look for the same main ideas across multiple lessons. [19] (vl9) 

29. My lessons are designed to explicitly support the broader goals within my curriculum: [20] 
(v20) 

30. 1 make sure that my tests measure the main ideas that I want students to master: [22] (v22) 

31.1 use specific techniques (e.g. outlines, graphic organizers, note taking) to make the main 
ideas obvious in my class. [23] (v23) 

32. 1 provide my students with multiple opportunities to practice using the main ideas of the 
curriculum in my content area. [24] (v24) 

The following questions were included to gather information about alignment: 

33. 1 primarily use a textbook to identify what is important to teach in my classes: [25] (v25) 

34. My curriculum, instruction, and assessment are aligned to the main ideas in the state 

standards. [27] (v27) 

35. 1 primarily use the state standards to identify what is important to teach in my classes: [26] 
(v26) 
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36. 1 modify the curriculum in my classes to match the state standards. [21] (v21) 

37. 1 am quite familiar with the main ideas in the state standards for my assigned content area(s). 
(28) 

38. 1 develop lesson plans to teach students the main ideas in the state standards. [29] (v29) 

39. 1 plan assessments to measure student mastery of main ideas relative to the state standards: 
[30] (v30) 

40. 1 use the state standards to identify what it important to assess in my classes. [31] (v31) 

The following questions were included to gather information about attitude towards state 

standards/ state testing. 

41. The state's expectations are reasonable for students in my assigned content area. [32] (v32) 

42. 1 think students at my school are doing as well as ever academically. [33] (v33) 

43. For students in my classes, improvement on state tests is extremely difficult to accomplish. 
[36] (v36) 

44. The state's expectations are unreasonable for students at my school. [37] (v37) 

45. 1 know how students at my school compare to students at other schools in my state on state 
tests. [38] (v38) 

46. 1 know how students at my school compare to students across the nation in terms of their 
performance on standardized tests. [39] (v39) 

47. State testing appropriately measures my students' strengths and weaknesses. [40] (v40) 

48. The statewide assessment addresses the main ideas of state content standards. [41] (v41) 

49. Please estimate the percentage of your students who will fail the state test in spite of 
everything you do. [42] 

The following questions were included to gather information about school improvement. 

50. Please select the TWO answer choices below that you think have the most potential to 
improve student achievement. [43] 

5 1 . If you selected OTHER in the previous question please describe the approaches that you 
think have the most potential to improve student achievement. [44] 

The following questions are included to gather information about the participants ’ degree of 

responding in a ‘socially desirable ’ way. 

52. 1 am always a good listener no matter whom I am talking to. [34] 

53. 1 am always willing to admit when I make a mistake. [35] 
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54. Appendix C: Survey Used in Pilot Study 


The actual survey sent to teachers, along with the email messages sent with the initial link to the 


survey and once the survey had been returned are included below: 

Email message sent with the link to the survey 

Thank you for your willingness to participate in this research effort. The survey you are about to 
take is part of an ongoing study aimed at better understanding teachers’ views on a variety of 
topics related to curriculum, instruction, and assessment. Your responses will provide valuable 
information about how teachers’ views differ across Oregon. By completing the survey, you are 
giving consent for us to use your responses in analyses that may result in publication in research 
journals. All identifying information will be removed from the data (your name, the name of 
your school, etc.) and your responses will remain anonymous. 

Based on feedback we received during our pilot study, we estimate you will need between 10 
and 20 minutes to complete the 53 question survey. Because the survey is delivered online, you 
will need to complete all 53 questions in once session, once you access the survey. To prevent 
people from submitting their responses more than once, the computer will not allow you to return 
to the survey once you have accessed it. 

Once you have responded to all the questions, your data will be sent automatically to a database, 
and I will receive notice that you have completed the survey. 

At that time, I will email you once again to get the information we need in order to send you a 
check for $10 to thank you for your time. 


Julie Alonzo 
Research Assistant 
Behavioral Research and Teaching 
University of Oregon 


Again, I sincerely thank you for your help. 



the purpose of the survey, the 
incentive offered for completing 
the survey, and instructions for 
completion. 
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CITES Survey Revised November 1, 2004 

The following survey is part of a research effort aimed at better understanding teachers’ views on 
a variety of topics related to curriculum, instruction, and assessment. Data from your responses 
will be compiled with other teachers’ responses for analysis. All identifying information (your 
name, the name of your school, etc.) will be removed from the data prior to analysis. 

If you are unable to provide a response to a particular question, simply skip that item and move 
on to the next. 

Note: Some questions were adapted from Woolfolk, A. & Hoy, W. (1990). Prospective teachers’ sense of efficacy 
and beliefs about control. Journal of Educational Psychology, 82(1), 81-91 or Crowne & Marlowe (1964) Social 
desirability scale, as cited in Ray, J. (1984). The reliability of short social desirability scales. The Journal of Social 
Psychology, 123, 133-134. 

1. Overall, I am comfortable teaching the curriculum in my assigned classes. 

strongly disagree 

disagree 

agree 

strongly agree 

2. I am well prepared to teach my assigned content classes. 

strongly disagree 

disagree 

agree 

strongly agree 

3. The amount a student can learn is primarily related to family background. 

strongly disagree 

disagree 

agree 

strongly agree 

4. If I try really hard, I can get through to even the most difficult or unmotivated students. 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 
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5. If a student in my class becomes disruptive and noisy, I know effective techniques to 
redirect him/her quickly. 

strongly disagree 

disagree 

agree 

strongly agree 

6. When one of my students gets a better grade than he/she typically gets, it is usually 
because I found a better way of teaching that student. 

strongly disagree 

disagree 

agree 

strongly agree 

7. If a student did not remember information I gave in a previous lesson, I would know how 
to increase his/her retention in the next lesson. 

strongly disagree 

disagree 

agree 

strongly agree 

8. When a student is having difficulty with an assignment, I am able to adjust it to his/her 
level of skill: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

9. When it comes right down to it, a student’s motivation and performance depends on 
his/her home environment, not on the teacher. 

strongly disagree 

disagree 

agree 

strongly agree 

10. Some students in my classes would be better placed with a teacher who has additional 
training. 

strongly disagree 

disagree 

agree 

strongly agree 
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11. I need additional support (e.g., classroom aides) to address the needs of all students in 
my classes. 

strongly disagree 

disagree 

agree 

strongly agree 

12. Having students with special needs in my classroom has negative consequences for other 
students. 

strongly disagree 

disagree 

agree 

strongly agree 

13. All my students can learn the main ideas in my curriculum. 

strongly disagree 

disagree 

agree 

strongly agree 

14. For students with special needs to succeed, I have to take instructional time away from 
other students. 

strongly disagree 

disagree 

agree 

strongly agree 

15. The main ideas of the curriculum in my content area(s) are addressed across the units in 
my class: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 
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16. 1 review the ‘big picture’ of my curriculum to ensure that there is coherence across 
lessons in my class: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

17. I teach my students to look for the same main ideas across multiple lessons. 

strongly disagree 

disagree 

agree 

strongly agree 

18. My lessons are designed to explicitly support the broader goals within my curriculum: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

19. 1 modify the curriculum in my classes to match the state standards: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

20. I make sure that my tests measure the main ideas that I want students to master: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

21. I use specific techniques (e.g., outlines, graphic organizers, note taking) to make the 
main ideas obvious in my class: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 
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22. 1 provide my students with multiple opportunities to practice using the main ideas of the 
curriculum in my content area: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

23. 1 primarily use a textbook to identify what is important to teach in my classes: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

24. 1 primarily use the state standards to identify what is important to teach in my classes: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

25. My curriculum, instruction, and assessments are aligned to the main ideas in the state 
standards: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

26. 1 am quite familiar with the main ideas in the state standards for my assigned content 
area(s). 

strongly disagree 

disagree 

agree 

strongly agree 

27. 1 develop lessons plans to teach students the main ideas in the state standards: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 
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28. 1 plan assessments to measure student mastery of main ideas relative to the state 
standards: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

29. 1 use the state standards to identify what is important to assess in my classes: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

30. The state’s expectations are reasonable for students in my assigned content area. 

strongly disagree 

disagree 

agree 

strongly agree 

31.1 think students at my school are doing as well as ever academically. 

strongly disagree 

disagree 

agree 

strongly agree 

32. 1 am always a good listener, no matter whom I am talking to. 

strongly disagree 

disagree 

agree 

strongly agree 

33. 1 am always willing to admit when I make a mistake. 

strongly disagree 

disagree 

agree 

strongly agree 
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34. For students in my classes, improvement on state tests is extremely difficult to 
accomplish. 

strongly disagree 

disagree 

agree 

strongly agree 

35. The state’s expectations are unreasonable for students at my school. 

strongly disagree 

disagree 

agree 

strongly agree 

36. 1 know how students at my school compare to students at other schools in my state on 
state tests. 

strongly disagree 

disagree 

agree 

strongly agree 

37. 1 know how students at my school compare to students across the nation in terms of their 
performance on standardized tests. 

strongly disagree 

disagree 

agree 

strongly agree 

38. State testing appropriately measures my students’ strengths and weaknesses. 

strongly disagree 

disagree 

agree 

strongly agree 

39. The statewide assessment addresses the main ideas of state content standards. 

strongly disagree 

disagree 

agree 

strongly agree 
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40. Please estimate the percentage of your students who will fail the state test in spite of 
everything you do. 

[text response] 

41. Please select the TWO answer choices below that you think have the most potential to 
improve student achievement. 

teacher staff development 
better curriculum planning 
improved materials selection 
include teachers in decisions 
mandate standardized curriculum 
let teachers control curriculum 
more frequent assessments 
other 

42. If you selected OTHER in the previous question, please describe the approaches that you 
think have the most potential to improve student achievement. 

[text response] 

Part II 

The following questions are included in order to give us demographic information that will aid in 
grouping responses for analysis. Your honest answers are very much appreciated. 

43. What is your gender? 

male 

female 

44. What is your age? 

[text response] 

45. What is your race/ethnicity? 

[text response] 

46. What is the highest degree you have earned? 

BA/BS 

MA/MS/MEd 

DEd/PhD 
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47. Including this year, how many years have you taught? 

[numeric response] 

48. What grade levels and content areas are you licensed to teach? 

[text response] 

49. What content area(s) are you teaching this year? 

[text response] 

50. What grade level(s) are you teaching this year? 

[text response] 

49. Approximately how many students attend your school? 

fewer than 250 
251 - 500 
501 - 1000 
1001 - 1500 
more than 1500 

5 1 . Approximately what percentage of students at your school receive free/reduced price 
lunch? 

less than 10% 

11 % - 20 % 

21% - 30% 

31% -40% 

41% -50% 
more than 50% 

52. How would you characterize your school’s setting? 
rural 

small town 

urban 

suburban 

53. Please feel free to share any additional thoughts you might have on topics related to 
education, (optional) 

[text response] 
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Email sent once teachers completed the survey: 

Thank you so much for participating in our study. The assistance of professional educators like 
you is vital for ongoing research. 

In order to process payment for your participation, I will need three pieces of information from 
you: 

1. Your name, as you would like it to appear on your check. 

2. The address where you would like the check to be mailed. 

3. Your social security number (required by the government for all payments made through 
grant sources). 

You may either send the information to me in an email or mail it through the US postal service if 
you are uncomfortable sending it over the internet. 

If you would prefer, you may also call me, Julie Alonzo, at the university (541) 346-1649 and 
give me your information over the phone. 

Checks will be sent from the Bethel School District. Once I have submitted the request for 
payment along with all required paperwork, it typically takes between 3 and 5 weeks for checks 
to be mailed. 

Please feel free to contact me if you have any questions. Again, thank you very much for helping 
out with this research. 

Sincerely, 

Julie Alonzo 
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Appendix D: Mplus Output Files for Measurement Model 

First, I ran the model using only the measured variables linked to the latent construct of 
‘efficacy’. 

INPUT INSTRUCTIONS 

Title: CITES Survey Analysis — Efficacy Factors — 2 factors 

Data: File is "C:\Julie\Mplus\CFA_Data.dat"; 

Variable: 

Names are vl v3 v4 v5 v6 v7 v8 v9 vlO vl 1 vl2 vl3 vl4 
vl5 vl6 vl7 vl8 vl9 v20 v21 v22 v23 v24 v25 v26 v27 v28 
v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41; 

Usevariables = v3-v6 v8; 

Missing are all (-9); 

Model: 

con BY v3@l v4; 
pers BY v5@lv6v8; 

Analysis: 

Estimator is ML; 

Iterations = 1000; 

Output: 

Techl mod Sampstat standardized; 

INPUT READING TERMINATED NORMALLY 

CITES Survey Analysis — Efficacy Factors — 2 factors 

SUMMARY OF ANALYSIS 
Number of groups 1 

Number of observations 101 

Number of y- variables 5 

Number of x- variables 0 

Number of continuous latent variables 2 

Observed variables in the analysis 
V3 V4 V5 V6 V8 

Continuous latent variables in the analysis 
CON PERS 

Estimator ML 

Maximum number of iterations 1000 
Convergence criterion 0.500D-04 

Input data file(s) 

C :\J ulie\Mplus\CFA_Data.dat 

Input data format FREE 

Mplus VERSION 2.02 PAGE 2 

CITES Survey Analysis — Efficacy Factors — 2 factors 


The technical 
features described 
in Appendix D likely 
exceed the steps 
one would take for 
most surveys used 
in school settings. 
They are included 
in this instance 
because this report 
is documenting 
evidence of 
construct validity 
related to the 
survey instrument 
as a measurement 
tool. 
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SAMPLE STATISTICS 


Means 

V3 V4 V5 V6 V8 


1 3.495 3.406 2.861 2.713 2.604 


Covariances 

V3 V4 V5 V6 V8 


V3 

0.652 




V4 

0.567 

0.684 



V5 

0.169 

0.107 

0.501 


V6 

0.134 

0.168 

0.230 

0.967 

V8 

0.108 

0.132 

0.105 

0.185 


Correlations 

V3 V4 V5 V6 V8 


V3 

1.000 




V4 

0.849 

1.000 



V5 

0.296 

0.183 

1.000 


V6 

0.168 

0.206 

0.330 

1.000 

V8 

0.222 

0.266 

0.246 

0.313 


THE MODEL ESTIMATION TERMINATED NORMALLY 

TESTS OF MODEL FIT 

Chi-Square Test of Model Fit 

Value 8.853 

Degrees of Freedom 4 
P-Value 0.0646 

Chi-Square Test of Model Fit for the Baseline Model 

Value 172.784 

Degrees of Freedom 10 

P- Value 0.0000 

CFI/TLI 

CFI 0.970 

TLI 0.925 

HO Value -503.287 

HI Value -498.861 

Information Criteria 

Number of Free Parameters 1 1 

Akaike (AIC) 1028.574 
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Bayesian (BIC) 1057.341 

Sample-Size Adjusted BIC 1022.598 
(n* = (n + 2) / 24) 

RMSEA (Root Mean Square Error Of Approximation) 

Estimate 0.110 

90 Percent C.I. 0.000 0.209 

Probability RMSEA <= .05 0.1 30 

SRMR (Standardized Root Mean Square Residual) 

Value 0.034 


MODEL RESULTS 



Estimates 

S.E. Est./S.E. 

Std StdYX 

CON 

BY 






V3 


1.000 

0.000 

0.000 

0.762 

0.948 

V4 


0.967 

0.175 

5.534 

0.737 

0.895 

PERS 

BY 






V5 


1.000 

0.000 

0.000 

0.393 

0.558 

V6 


1.409 

0.482 

2.921 

0.554 

0.566 

V8 


0.781 

0.272 

2.874 

0.307 

0.513 

PERS 

WITH 





CON 


0.132 

0.051 

2.597 

0.441 

0.441 

Residual Variances 





V3 


0.065 

0.100 

0.652 

0.065 

0.101 

V4 


0.134 

0.095 

1.414 

0.134 

0.198 

V5 


0.341 

0.071 

4.828 

0.341 

0.688 

V6 


0.651 

0.137 

4.737 

0.651 

0.680 

V8 


0.264 

0.050 

5.322 

0.264 

0.737 

Variances 






CON 


0.581 

0.135 

4.319 

1.000 

1.000 

PERS 


0.154 

0.072 

2.132 

1.000 

1.000 


R-SQUARE 

Observed 

Variable 

R-Square 

V3 

0.899 

V4 

0.802 

V5 

0.312 

V6 

0.320 

V8 

0.263 


MODEL MODIFICATION INDICES 

Minimum M.I. value for printing the modification index 10.000 
M.I. E.P.C. StdE.P.C. StdYX E.P.C. 


6/29/05 



CITES Survey Development and Validation 


63 


No modification indices above the minimum value. 


TECHNICAL 1 OUTPUT 
PARAMETER SPECIFICATION 
LAMBDA 
CON PERS 


V3 

0 

0 




V4 

1 

0 




V5 

0 

0 




V6 

0 

2 




V8 

0 

3 





THETA 






V3 

V4 

V5 

V6 

V8 

V3 

4 





V4 

0 

5 




V5 

0 

0 

6 



V6 

0 

0 

0 

7 


V8 

0 

0 

0 

0 

8 



PSI 





CON 

PERS 



CON 

9 




PERS 

10 

11 



STARTING VALUES 




LAMBDA 





CON 

PERS 



V3 

1.000 

0.000 



V4 

1.000 

0.000 



V5 

0.000 

1.000 



V6 

0.000 

1.000 



V8 

0.000 

1.000 




THETA 





V3 

V4 V5 

V6 

V8 

V3 

0.326 




V4 

0.000 

0.342 



V5 

0.000 

0.000 

0.250 


V6 

0.000 

0.000 

0.000 

0.483 

V8 

0.000 

0.000 

0.000 

0.000 


PSI 





CON 

PERS 



CON 

0.050 




PERS 

0.000 

0.050 




0.181 
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Next, I ran the model using the measured variables theoretically linked to the underlying latent 
construct of ‘alignment.’ 

INPUT INSTRUCTIONS 

Title: CITES Survey Analysis — Alignment 

Data: File is "C:\Julie\Mplus\CFA_Data.dat"; 

Variable: 

Names are vl v3 v4 v5 v6 v7 v8 v9 vlO vll vl2 vl3 vl4 
vl5 vl6 vl7 vl8 vl9 v20 v21 v22 v23 v24 v25 v26 v27 v28 
v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41; 

Usevariables = v21 v26 v27 v29-v31; 

Missing are all (-9); 

Model: 

align BY v29@l v21 v26 v27 v30 v31; 
v29 WITH v27; 

Analysis: 

Estimator is ME; 

Iterations = 1000; 

Output; 

Techl mod Sampstat standardized; 

INPUT READING TERMINATED NORMALLY 
CITES Survey Analysis — Alignment 

SUMMARY OF ANALYSIS 
Number of groups 1 

Number of observations 95 

Number of y-variables 6 

Number of x-variables 0 

Number of continuous latent variables 1 

Observed variables in the analysis 
V21 V26 V27 V29 V30 V31 

Continuous latent variables in the analysis 
ALIGN 

Estimator ML 

Maximum number of iterations 1000 
Convergence criterion 0.500D-04 


SAMPLE STATISTICS 
Means 

V21 V26 V27 V29 V30 


1 2.863 2.737 3.295 2.989 2.789 


Means 

V31 


1 2.705 
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Covariances 



V21 

V26 

V27 

V29 

V30 

V21 

1.098 





V26 

0.783 

1.175 




V27 

0.413 

0.408 

0.614 



V29 

0.647 

0.742 

0.514 

0.947 


V30 

0.567 

0.710 

0.403 

0.721 

0.891 

V31 

0.693 

0.890 

0.407 

0.720 

0.746 


Covariances 

V31 


V31 1.019 


Correlations 



V21 

V26 

V27 

V29 

V30 

V21 

1.000 





V26 

0.689 

1.000 




V27 

0.503 

0.480 

1.000 



V29 

0.635 

0.704 

0.674 

1.000 


V30 

0.573 

0.694 

0.545 

0.785 

1.000 

V31 

0.656 

0.813 

0.514 

0.733 

0.783 


Correlations 

V31 


V31 1.000 

THE MODEL ESTIMATION TERMINATED NORMALLY 
TESTS OF MODEL FIT 


Chi-Square Test of Model Fit 

Value 19.239 

Degrees of Freedom 8 
P- Value 0.0136 


Chi-Square Test of Model Fit for the Baseline Model 


Value 431.625 

Degrees of Freedom 15 
P- Value 0.0000 

CFI/TLI 

CFI 0.973 

TLI 0.949 

Loglikelihood 

HO Value -581.347 

HI Value -571.728 
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Information Criteria 

Number of Free Parameters 13 
Akaike(AIC) 1188.695 

Bayesian (BIC) 1221.895 

Sample-Size Adjusted BIC 1 180.851 

(n* = (n + 2) / 24) 

RMSEA (Root Mean Square Error Of Approximation) 
Estimate 0.122 

90 Percent C.I. 0.052 0.192 

Probability RMSEA <= .05 0.047 

SRMR (Standardized Root Mean Square Residual) 
Value 0.033 


MODEL RESULTS 


Estimates 

S.E. Est./S.E. 

Std StdYX 

ALIGN BY 

V29 

1.000 

0.000 

0.000 

0.812 

0.839 

V21 

0.944 

0.114 

8.282 

0.767 

0.736 

V26 

1.151 

0.108 

10.665 

0.935 

0.867 

V27 

0.567 

0.076 

7.477 

0.461 

0.591 

V30 

0.984 

0.095 

10.349 

0.800 

0.851 

V31 

1.124 

0.098 

11.530 

0.914 

0.910 

V29 WITH 

V27 

0.134 

0.042 

3.199 

0.134 

0.177 

Residual Variances 
V21 0.499 

0.079 

6.305 

0.499 

0.459 

V26 

0.288 

0.054 

5.338 

0.288 

0.248 

V27 

0.395 

0.060 

6.585 

0.395 

0.650 

V29 

0.277 

0.049 

5.681 

0.277 

0.295 

V30 

0.243 

0.044 

5.550 

0.243 

0.275 

V31 

0.173 

0.039 

4.432 

0.173 

0.172 

Variances 

ALIGN 

0.660 

i 0.133 

4.972 1.00C 

I 1.000 

R-SQUARE 


Observed 

Variable R-Square 
V21 0.541 

V26 0.752 

V27 0.350 

V29 0.705 

V30 0.725 

V31 0.828 

MODEL MODIFICATION INDICES 


Minimum M.I. value for printing the modification index 10.000 
M.I. E.P.C. StdE.P.C. StdYX E.P.C. 

No modification indices above the minimum value. 

TECHNICAL 1 OUTPUT 
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PARAMETER SPECIFICATION 

LAMBDA 

ALIGN 


V21 1 

V26 2 

V27 3 

V29 0 

V30 4 

V31 5 

THETA 

V21 V26 V27 V29 V30 


V21 6 

V26 0 

V27 0 

V29 0 

V30 0 

V31 0 


THETA 

V31 


V31 12 

PSI 

ALIGN 


ALIGN 13 

STARTING VALUES 


LAMBDA 

ALIGN 


V21 

1.000 





V26 

1.000 





V27 

1.000 





V29 

1.000 





V30 

1.000 





V31 

1.000 






THETA 






V21 

V26 

V27 

V29 

V30 

V21 

0.549 





V26 

0.000 

0.587 




V27 

0.000 

0.000 

0.307 



V29 

0.000 

0.000 

0.000 

0.473 


V30 

0.000 

0.000 

0.000 

0.000 

0.446 

V31 

0.000 

0.000 

0.000 

0.000 

0.000 


THETA 


7 

0 8 

0 9 10 

0 0 0 11 

0 0 0 0 
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V31 


V31 0.509 


PSI 

ALIGN 


ALIGN 0.050 

Then, I ran the model using the measured variables linked to the two underlying latent variables 
theoretically measuring ‘organization.’ 

INPUT INSTRUCTIONS 

Title: CITES Survey Analysis — Organization, Revised 

Data: File is "C:\Julie\Mplus\CFA_Data.dat"; 

Variable: 

Names are vl v3 v4 v5 v6 v7 v8 v9 vlO vll vl2 vl3 vl4 
vl5 vl6 vl7 vl8 vl9 v20 v21 v22 v23 v24 v25 v26 v27 v28 
v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41; 

Usevariables = vl7-v20 v23-v24; 

Missing are all (-9); 


Model: 


less 

BY 

vl8@l vl7 vl9 v20; 

main 

BY 

v24@l v23; 

vl7 

WITH 

vl8; 

vl9 

WITH 

v20; 


Analysis: 

Estimator is ML; 

Iterations = 1000; 

Output; 

Techl mod Sampstat standardized; 

INPUT READING TERMINATED NORMALLY 

CITES Survey Analysis — Organization, Revised 
SUMMARY OF ANALYSIS 


Number of groups 1 

Number of observations 93 

Number of y-variables 6 

Number of x-variables 0 

Number of continuous latent variables 2 

Observed variables in the analysis 
V17 V18 V19 V20 V23 V24 


Continuous latent variables in the analysis 
LESS MAIN 

Estimator ML 
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Maximum number of iterations 1000 
Convergence criterion 0.500D-04 

SAMPLE STATISTICS 


Means 

V 17 V18 V19 V20 V23 


1 3.419 3.226 3.194 3.376 3.398 

Means 

V24 


1 3.548 

Covariances 



V 17 

V18 

V19 

V20 

V23 

V17 

0.550 





V18 

0.383 

0.872 




V19 

0.124 

0.206 

0.397 



V20 

0.177 

0.425 

0.231 

0.563 


V23 

0.081 

0.246 

0.031 

0.196 

0.633 

V24 

0.061 

0.157 

0.067 

0.161 

0.247 


Covariances 

V24 


V24 0.511 


Correlations 



V 17 

V18 

V19 

V20 

V23 

V17 

1.000 





V18 

0.552 

1.000 




V19 

0.266 

0.350 

1.000 



V20 

0.319 

0.606 

0.488 

1.000 


V23 

0.138 

0.331 

0.062 

0.329 

1.000 

V24 

0.115 

0.236 

0.148 

0.300 

0.434 


Correlations 

V24 


V24 1.000 


THE MODEL ESTIMATION TERMINATED NORMALLY 

TESTS OF MODEL FIT 

Chi-Square Test of Model Fit 

Value 4.197 

Degrees of Freedom 6 
P- Value 0.6499 
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Chi-Square Test of Model Fit for the Baseline Model 

Value 141.737 

Degrees of Freedom 15 
P- Value 0.0000 

CFI/TLI 

CFI 1.000 

TLI 1.036 

Loglikelihood 

HO Value -563.794 

HI Value -561.695 

Information Criteria 

Number of Free Parameters 15 

Akaike(AIC) 1157.587 

Bayesian (BIC) 1195.576 

Sample-Size Adjusted BIC 1 148.225 

(n* = (n + 2) / 24) 

RMSEA (Root Mean Square Error Of Approximation) 
Estimate 0.000 

90 Percent C.I. 0.000 0.109 

Probability RMSEA <= .05 0.762 

SRMR (Standardized Root Mean Square Residual) 
Value 0.036 

MODEL RESULTS 


Estimates S.E. Est./S.E. Std StdYX 
LESS BY 


V18 


1.000 

0.000 

0.000 

0.702 

0.756 

V17 


0.413 

0.111 

3.741 

0.290 

0.393 

V19 


0.376 

0.140 

2.695 

0.264 

0.422 

V20 


0.854 

0.220 

3.873 

0.600 

0.803 

MAIN 

BY 






V24 


1.000 

0.000 

0.000 

0.427 

0.600 

V23 


1.342 

0.452 

2.966 

0.572 

0.723 

MAIN 

WITH 





LESS 


0.173 

0.067 

2.579 

0.579 

0.579 

V17 

WITH 






V18 


0.174 

0.075 

2.335 

0.174 

0.255 

V19 

WITH 






V20 


0.070 

0.053 

1.311 

0.070 

0.149 

Residual Variances 





V17 


0.460 

0.075 

6.124 

0.460 

0.845 

V18 


0.370 

0.130 

2.849 

0.370 

0.428 

V19 


0.323 

0.055 

5.841 

0.323 

0.822 

V20 


0.198 

0.091 

2.178 

0.198 

0.355 
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V23 

0.299 

0.114 

2.634 

0.299 

0.477 

V24 

0.324 

0.075 

4.311 

0.324 

0.640 

Variances 

LESS 

0.493 

0.164 

3.004 

1.000 

1.000 

MAIN 

0.182 

0.081 

2.235 

1.000 

1.000 


R-SQUARE 

Observed 

Variable R-Square 
V17 0.155 

V18 0.572 

V19 0.178 

V20 0.645 

V23 0.523 

V24 0.360 

MODEL MODIFICATION INDICES 

Minimum M.I. value for printing the modification index 10.000 
M.I. E.P.C. StdE.P.C. StdYX E.P.C. 

No modification indices above the minimum value. 


TECHNICAL 1 OUTPUT 

PARAMETER SPECIFICATION 
LAMBDA 
LESS MAIN 


V17 

1 

0 



V18 

0 

0 



V19 

2 

0 



V20 

3 

0 



V23 

0 

4 



V24 

0 

0 




THETA 





V 17 

V18 

V19 

V20 

V17 

5 




V18 

6 

7 



V19 

0 

0 

8 


V20 

0 

0 

9 

10 

V23 

0 

0 

0 

0 

V24 

0 

0 

0 

0 


THETA 





V24 




V24 

12 





PSI 





LESS 

MAIN 




LESS 13 
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MAIN 14 15 

STARTING VALUES 
LAMBDA 



LESS 

MAIN 




V17 

1.000 

0.000 




V18 

1.000 

0.000 




V19 

1.000 

0.000 




V20 

1.000 

0.000 




V23 

0.000 

1.000 




V24 

0.000 

1.000 





THETA 






V 17 

V18 

V19 

V20 

V23 

V17 

0.275 





V18 

0.000 

0.436 




V 19 

0.000 

0.000 

0.198 



V20 

0.000 

0.000 

0.000 

0.282 


V23 

0.000 

0.000 

0.000 

0.000 

0.317 

V24 

0.000 

0.000 

0.000 

0.000 

0.000 


THETA 

V24 


V24 0.256 

PSI 

LESS MAIN 


LESS 0.050 

MAIN 0.000 0.050 

I then ran the model using only the four measured variables associated with the latent construct 
‘inclusivity.’ 

INPUT INSTRUCTIONS 
Title: CITES Survey Analysis — Inclusivity, 

Data: File is "C:\Julie\Mplus\CFA_Data.dat"; 

Variable: 

Names are vl v3 v4 v5 v6 v7 v8 v9 vlO vll vl2 vl3 vl4 
vl5 vl6 vl7 vl8 vl9 v20 v21 v22 v23 v24 v25 v26 v27 v28 
v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41; 

Usevariables = vl2-vl4 vl6; 

Missing are all (-9); 

Model: 

inclu BY vl4@l vl2 vl3 vl6; 

Analysis: 

Estimator is ML; 

Iterations = 1000; 

Output; 

Techl mod Sampstat standardized; 
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INPUT READING TERMINATED NORMALLY 
CITES Survey Analysis — Inclusivity, 

SUMMARY OF ANALYSIS 

Number of groups 1 

N umber of observations 101 

Number of y-variables 4 

Number of x-variables 0 

Number of continuous latent variables 1 

Observed variables in the analysis 
V12 V 13 V 14 V16 

Continuous latent variables in the analysis 
INCLU 

Estimator ML 

Maximum number of iterations 1000 
Convergence criterion 0.500D-04 

SAMPLE STATISTICS 
Means 



V 12 

V13 

V14 

V16 

1 

2.416 

2.158 

2.861 

2.337 


Covariances 




V 12 

V13 

V14 

V16 

V 12 

0.725 




V13 

0.153 

0.575 



V 14 

0.168 

0.202 

0.541 


V16 

0.089 

0.196 

0.287 

0.546 


Correlations 




V 12 

V13 

V14 

V16 

V 12 

1.000 




V13 

0.238 

1.000 



V 14 

0.269 

0.363 

1.000 


V 16 

0.141 

0.350 

0.529 

1.000 


THE MODEL ESTIMATION TERMINATED NORMALLY 

TESTS OF MODEL FIT 
Chi-Square Test of Model Fit 

Value 2.562 

Degrees of Freedom 2 

P- Value 0.2744 

Chi-Square Test of Model Fit for the Baseline Model 
Value 61.681 

Degrees of Freedom 6 

P- Value 0.0000 
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CFI/TLI 

CFI 0.990 

TLI 0.970 

Loglikelihood 

HO Value -435.826 

HI Value -434.545 


Information Criteria 

Number of Free Parameters 8 
Akaike (AIC) 887.651 

Bayesian (BIC) 908.572 

Sample-Size Adjusted BIC 883.305 
(n* = (n + 2) / 24) 


RMSEA (Root Mean Square Error Of Approximation) 
Estimate 0.053 

90 Percent C.I. 0.000 0.212 

Probability RMSEA <= .05 0.364 


SRMR (Standardized Root Mean Square Residual) 
Value 0.036 


MODEL RESULTS 

Estimates S.E. Est./S.E. Std StdYX 


INCLU BY 

V14 

1.000 

0.000 

0.000 

0.569 

0.778 

V12 

0.477 

0.183 

2.609 

0.272 

0.321 

V13 

0.660 

0.178 

3.713 

0.376 

0.498 

V16 

0.865 

0.212 

4.080 

0.492 

0.670 

Residual Variances 
V12 0.644 

0.095 

6.805 

0.644 

0.897 

V13 

0.428 

0.069 

6.180 

0.428 

0.752 

V14 

0.211 

0.077 

2.733 

0.211 

0.395 

V16 

0.298 

0.068 

4.353 

0.298 

0.551 

Variances 

INCLU 

0.324 

0.099 

3.258 

1.000 

1.000 


R-SQUARE 

Observed 

Variable R-Square 
V12 0.103 

V13 0.248 

V14 0.605 

V16 0.449 

MODEL MODIFICATION INDICES 

Minimum M.I. value for printing the modification index 10.000 
M.I. E.P.C. StdE.P.C. StdYX E.P.C. 

No modification indices above the minimum value. 


TECHNICAL 1 OUTPUT 
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PARAMETER SPECIFICATION 
LAMBDA 
INCLU 

V 12 1 

V13 2 

V 14 0 

V 16 3 

THETA 

V 12 V13 V14 V16 

V 12 4 

V13 0 5 

V 14 0 0 6 

V 16 0 0 0 7 

PSI 

INCLU 

INCLU 8 

STARTING VALUES 


LAMBDA 

INCLU 


V 12 

1.000 



V13 

1.000 



V 14 

1.000 



V 16 

1.000 




THETA 




V 12 

V13 

V14 V16 

V 12 

0.363 



V13 

0.000 

0.287 


V 14 

0.000 

0.000 

0.270 

V16 

0.000 

0.000 

0.000 0.273 


PSI 

INCLU 

INCLU 0.050 

Finally, I combined all four of the previous ‘mini models’ into a single model to test the four 
constructs measured together. 

INPUT INSTRUCTIONS 

Title: CITES Survey Analysis — Efficacy (2 factor), 

Organization (2 factor). Alignment, Inclusivity 
Data: File is "C:\Julie\Mplus\CFA_Data.dat"; 

Variable: 

Names are vl v3 v4 v5 v6 v7 v8 v9 vlO vll vl2 vl3 vl4 
v!5 vl6 v!7 v!8 vl9 v20 v21 v22 v23 v24 v25 v26 v27 v28 
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v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41; 

Usevariables = v3-v6 v8 vl2-vl4 vl6-v21 v23-v24 v26 v27 v29-v31; 

Missing are all (-9); 

Model: 

con BY v3@l v4; 
pers BY v5@lv6v8; 

less BY vl8@l vl7 vl9v20; 

main BY v24@lv23; 

align BY v29@ 1 v2 1 v26 v27 v30 v3 1 ; 

inclu BY vl4@l vl2 vl3 vl6; 

vl7 WITH vl8; 
v 19 WITH v20; 
v29 WITH v27; 

Analysis: 

Estimator is ML; 

Iterations = 1000; 

Output: 

Techl mod Sampstat standardized; 

INPUT READING TERMINATED NORMALLY 

CITES Survey Analysis — Efficacy (2 factor). 

Organization (2 factor), Alignment, Inclusivity 
SUMMARY OF ANALYSIS 
Number of groups 1 

Number of observations 85 

Number of y- variables 21 

Number of x-variables 0 

Number of continuous latent variables 6 


Observed variables in the analysis 


V3 

V4 

V5 

V6 

V8 

V12 

V13 

V 14 

V16 

V17 

V18 

V19 

V20 

V21 

V23 

V24 

V26 

V27 

V29 

V30 

V31 





Continuous latent variables in the analysis 
CON PERS LESS MAIN ALIGN INCLU 
Estimator ML 

Maximum number of iterations 1000 

Convergence criterion 0.500D-04 

SAMPLE STATISTICS 
Means 


V3 

V4 

V5 

V6 

V8 

3.494 

3.400 

2.847 

2.659 

2.635 

Means 

V 12 

V13 

V14 

V16 

V17 

2.365 

2.141 

2.812 

2.282 

3.388 

Means 

V18 

V19 

V20 

V21 

V23 

3.212 

3.212 

3.353 

2.765 

3.376 
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Means 

V24 

V26 

V27 

V29 

V30 

1 

3.529 

2.635 

3.259 

2.941 

2.776 


Means 






V31 





1 

2.647 






Covariances 






V3 

V4 

V5 V6 V8 

V3 

0.658 





V4 

0.574 

0.695 




V5 

0.184 

0.121 

0.464 



V6 

0.135 

0.174 

0.173 

0.966 


V8 

0.063 

0.100 

0.110 

0.184 

0.354 

V 12 

0.044 

0.043 

0.128 

0.221 

0.111 

V13 

0.013 

-0.021 

0.069 

0.120 

0.016 

V 14 

0.011 

-0.007 

0.102 

0.209 

0.073 

V16 

0.061 

0.064 

0.068 

0.228 

0.033 

V17 

0.104 

0.140 

0.012 

0.015 

0.060 

V18 

0.108 

0.176 

0.021 

0.061 

0.102 

V19 

0.096 

0.093 

0.033 

0.073 

0.066 

V20 

0.014 

0.036 

0.031 

0.110 

0.011 

V21 

0.153 

0.119 

0.190 

0.300 

0.151 

V23 

0.026 

0.038 

0.082 

0.166 

0.079 

V24 

0.069 

0.071 

0.153 

0.135 

0.029 

V26 

0.051 

-0.007 

0.182 

0.279 

0.199 

V27 

0.073 

0.098 

0.183 

0.077 

0.107 

V29 

-0.030 

0.012 

0.122 

0.265 

0.157 

V30 

0.028 

0.043 

0.156 

0.280 

0.156 

V31 

0.046 

0.048 

0.124 

0.295 

0.167 


Covariances 






V 12 

V13 

V14 

V16 

V 17 

V 12 

0.734 





V13 

0.138 

0.551 




V 14 

0.177 

0.158 

0.512 



V 16 

0.110 

0.150 

0.232 

0.443 


V17 

0.047 

-0.067 

-0.033 

-0.004 

0.550 

V18 

-0.019 

0.029 

0.052 

0.035 

0.429 

V19 

0.100 

0.005 

0.052 

-0.013 

0.143 

V20 

0.120 

0.104 

0.032 

-0.018 

0.195 

V21 

-0.068 

-0.121 

-0.033 

0.091 

0.069 

V23 

-0.020 

-0.018 

0.012 

-0.024 

0.090 

V24 

0.138 

0.043 

0.077 

-0.008 

0.066 

V26 

-0.032 

-0.103 

-0.058 

0.092 

-0.011 

V27 

-0.072 

-0.025 

-0.058 

0.033 

0.077 

V29 

-0.050 

-0.075 

-0.035 

0.064 

-0.036 

V30 

-0.001 

0.008 

0.017 

0.111 

-0.055 

V31 

0.059 

-0.021 

-0.032 

0.089 

-0.099 
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Covariances 



V 18 

V19 

V20 

V21 

V23 

V18 

0.859 





V19 

0.228 

0.407 




V20 

0.401 

0.258 

0.564 



V21 

0.181 

0.086 

0.167 

1.111 


V23 

0.217 

0.038 

0.175 

0.244 

0.642 

V24 

0.137 

0.077 

0.144 

0.138 

0.251 

V26 

0.090 

0.114 

0.118 

0.782 

0.294 

V27 

0.147 

0.064 

0.217 

0.419 

0.270 

V29 

0.096 

0.048 

0.152 

0.665 

0.332 

V30 

0.096 

0.036 

0.139 

0.625 

0.311 

V31 

0.052 

0.028 

0.138 

0.714 

0.361 


Covariances 






V24 

V26 

V27 

V29 

V30 

V24 

0.538 





V26 

0.136 

1.187 




V27 

0.111 

0.405 

0.646 



V29 

0.008 

0.764 

0.539 

0.985 


V30 

0.108 

0.775 

0.439 

0.796 

0.961 

V31 

0.070 

0.905 

0.426 

0.753 

0.813 


Covariances 






V31 





V31 

1.041 






Correlations 






V3 

V4 

V5 V6 V8 

V3 

1.000 





V4 

0.849 

1.000 




V5 

0.332 

0.214 

1.000 



V6 

0.169 

0.212 

0.259 

1.000 


V8 

0.131 

0.202 

0.272 

0.314 

1.000 

V 12 

0.063 

0.060 

0.219 

0.263 

0.217 

V13 

0.021 

-0.035 

0.137 

0.165 

0.037 

V 14 

0.019 

-0.012 

0.209 

0.297 

0.173 

V 16 

0.113 

0.116 

0.149 

0.349 

0.083 

V17 

0.172 

0.227 

0.025 

0.021 

0.136 

V18 

0.144 

0.228 

0.033 

0.067 

0.185 

V19 

0.187 

0.175 

0.075 

0.117 

0.175 

V20 

0.023 

0.057 

0.060 

0.149 

0.025 

V21 

0.179 

0.135 

0.264 

0.289 

0.241 

V23 

0.040 

0.057 

0.150 

0.210 

0.167 

V24 

0.115 

0.117 

0.307 

0.188 

0.066 

V26 

0.058 

-0.008 

0.245 

0.260 

0.307 

V27 

0.112 

0.146 

0.334 

0.098 

0.225 

V29 

-0.037 

0.014 

0.180 

0.272 

0.266 

V30 

0.036 

0.052 

0.233 

0.291 

0.267 

V31 

0.055 

0.056 

0.178 

0.294 

0.276 


Correlations 
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V 12 

V13 

V14 

V16 

V17 

V 12 

1.000 





V13 

0.217 

1.000 




V 14 

0.288 

0.297 

1.000 



V 16 

0.193 

0.304 

0.488 

1.000 


V17 

0.074 

-0.122 

-0.063 

-0.008 

1.000 

V18 

-0.023 

0.043 

0.079 

0.056 

0.624 

V19 

0.184 

0.012 

0.114 

-0.030 

0.302 

V20 

0.186 

0.187 

0.059 

-0.035 

0.349 

V21 

-0.075 

-0.155 

-0.044 

0.130 

0.088 

V23 

-0.029 

-0.030 

0.021 

-0.045 

0.152 

V24 

0.220 

0.080 

0.147 

-0.017 

0.121 

V26 

-0.034 

-0.127 

-0.074 

0.127 

-0.014 

V27 

-0.104 

-0.042 

-0.101 

0.062 

0.129 

V29 

-0.058 

-0.102 

-0.049 

0.098 

-0.049 

V30 

-0.001 

0.011 

0.024 

0.171 

-0.076 

V31 

0.067 

-0.028 

-0.043 

0.131 

-0.131 


Correlations 






V 18 

V19 

V20 

V21 

V23 

V18 

1.000 





V19 

0.386 

1.000 




V20 

0.575 

0.538 

1.000 



V21 

0.186 

0.128 

0.211 

1.000 


V23 

0.292 

0.075 

0.291 

0.289 

1.000 

V24 

0.201 

0.165 

0.262 

0.178 

0.427 

V26 

0.089 

0.164 

0.145 

0.681 

0.336 

V27 

0.197 

0.124 

0.359 

0.494 

0.420 

V29 

0.104 

0.076 

0.204 

0.636 

0.417 

V30 

0.105 

0.058 

0.189 

0.605 

0.396 

V31 

0.055 

0.043 

0.180 

0.664 

0.441 


Correlations 






V24 

V26 

V27 

V29 

V30 

V24 

1.000 





V26 

0.170 

1.000 




V27 

0.189 

0.462 

1.000 



V29 

0.011 

0.707 

0.676 

1.000 


V30 

0.150 

0.725 

0.557 

0.818 

1.000 

V31 

0.094 

0.815 

0.519 

0.744 

0.813 


Correlations 

V31 


V31 1.000 

THE MODEL ESTIMATION TERMINATED NORMALLY 

TESTS OF MODEL FIT 
Chi-Square Test of Model Fit 

Value 238.071 

Degrees of Freedom 171 

P- Value 0.0005 
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Chi-Square Test of Model Fit for the Baseline Model 
Value 990.484 

Degrees of Freedom 2 1 0 

P- Value 0.0000 

CFI/TLI 

CFI 0.914 

TLI 0.894 

Loglikelihood 

HO Value -1786.024 

HI Value -1666.989 


Information Criteria 

Number of Free Parameters 60 

Akaike (AIC) 3692.049 

Bayesian (BIC) 3838.608 

Sample-Size Adjusted BIC 3649.320 

(n* = (n + 2) / 24) 


RMSEA (Root Mean Square Error Of Approximation) 
Estimate 0.068 

90 Percent C.I. 0.046 0.088 

Probability RMSEA <= .05 0.086 

SRMR (Standardized Root Mean Square Residual) 

Value 0.075 


MODEL RESULTS 



Estimates 

S.E. Est./S.E. 

Std StdYX 

CON 

BY 






V3 


1.000 

0.000 

0.000 

0.677 

0.840 

V4 


1.236 

0.237 

5.218 

0.837 

1.010 

PERS 

BY 






V5 


1.000 

0.000 

0.000 

0.318 

0.469 

V6 


1.949 

0.558 

3.490 

0.619 

0.633 

V8 


0.894 

0.295 

3.030 

0.284 

0.480 

LESS 

BY 






V18 


1.000 

0.000 

0.000 

0.794 

0.862 

V17 


0.497 

0.110 

4.537 

0.395 

0.536 

V19 


0.354 

0.134 

2.641 

0.281 

0.443 

V20 


0.628 

0.198 

3.176 

0.499 

0.668 

MAIN 

BY 






V24 


1.000 

0.000 

0.000 

0.322 

0.442 

V23 


2.384 

0.986 

2.417 

0.768 

0.965 

ALIGN 

BY 






V29 


1.000 

0.000 

0.000 

0.843 

0.855 

V21 


0.912 

0.114 

8.006 

0.769 

0.734 
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V26 

1.104 

0.107 

10.349 

0.931 

0.859 

V27 

0.566 

0.079 

7.192 

0.477 

0.597 

V30 

1.026 

0.094 

10.973 

0.865 

0.888 

V31 

1.096 

0.095 

11.510 

0.924 

0.911 

INCLU 

BY 





V14 

1.000 

0.000 

0.000 

0.523 

0.736 

V12 

0.700 

0.223 

3.137 

0.366 

0.430 

V13 

0.625 

0.194 

3.219 

0.327 

0.443 

V16 

0.774 

0.192 

4.023 

0.405 

0.612 

PERS 

WITH 





CON 

0.081 

0.040 

2.027 

0.376 

0.376 

LESS 

WITH 





CON 

0.125 

0.072 

1.746 

0.232 

0.232 

PERS 

0.053 

0.043 

1.228 

0.211 

0.211 

MAIN 

WITH 





CON 

0.014 

0.025 

0.544 

0.063 

0.063 

PERS 

0.036 

0.023 

1.574 

0.353 

0.353 

LESS 

0.095 

0.052 

1.831 

0.371 

0.371 

ALIGN 

WITH 





CON 

0.028 

0.064 

0.442 

0.049 

0.049 

PERS 

0.147 

0.054 

2.726 

0.549 

0.549 

LESS 

0.112 

0.086 

1.304 

0.167 

0.167 

MAIN 

0.127 

0.062 

2.030 

i 0.467 

0.467 

INCLU 

WITH 





CON 

0.016 

0.046 

0.342 

0.044 

0.044 

PERS 

0.107 

0.040 

2.679 

0.646 

0.646 

LESS 

0.049 

0.062 

0.801 

0.119 

0.119 

MAIN 

-0.002 0.023 

-0.092 -0.012 -0.012 

ALIGN 0.009 0.059 

0.148 0.020 0.020 

V17 WITH 





V18 

0.110 

0.109 

1.011 

0.110 

0.162 

V19 WITH 





V20 

0.114 

0.056 

2.032 

0.114 

0.241 

V29 WITH 





V27 

0.131 

0.044 

2.986 

0.131 

0.166 

Residual Variances 





V3 

0.191 

0.087 

2.197 

0.191 

0.294 

V4 

-0.014 

0.125 

-0.112 

-0.014 

-0.020 

V5 

0.358 

0.061 

5.909 

0.358 

0.780 

V6 

0.571 

0.120 

4.752 

0.571 

0.599 

V8 

0.269 

0.046 

5.862 

0.269 

0.769 

V12 

0.592 

0.099 

5.949 

0.592 

0.815 

V13 

0.438 

0.074 

5.902 

0.438 

0.804 

V14 

0.232 

0.067 

3.487 

0.232 

0.459 

V16 

0.274 

0.056 

4.922 

0.274 

0.626 

V17 

0.387 

0.089 

4.359 

0.387 

0.713 
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V18 

0.218 

0.187 

1.167 

0.218 

0.257 

V19 

0.323 

0.057 

5.658 

0.323 

0.803 

V20 

0.309 

0.087 

3.549 

0.309 

0.554 

V21 

0.507 

0.084 

6.031 

0.507 

0.462 

V23 

0.044 

0.209 

0.211 

0.044 

0.070 

V24 

0.428 

0.075 

5.684 

0.428 

0.804 

V26 

0.307 

0.058 

5.315 

0.307 

0.262 

V27 

0.411 

0.066 

6.252 

0.411 

0.644 

V29 

0.262 

0.049 

5.363 

0.262 

0.269 

V30 

0.202 

0.041 

4.922 

0.202 

0.212 

V31 

0.175 

0.040 

4.421 

0.175 

0.170 

Variances 

CON 

0.459 

0.126 

3.649 

1.000 

1.000 

PERS 

0.101 

0.051 

1.980 

1.000 

1.000 

LESS 

0.631 

0.223 

2.829 

1.000 

1.000 

MAIN 

0.104 

0.061 

1.708 

1.000 

1.000 

ALIGN 

0.711 

0.146 

4.856 

i 1.000 

> 1.000 

INCLU 0.274 

R-SQUARE 

Observed 

Variable R-Square 
V3 0.706 

0.089 

3.076 

1.000 

1.000 


V4 Undefined 0.10204E+01 

V5 0.220 

V6 0.401 

V8 0.231 

V12 0.185 

V13 0.196 

V14 0.541 

V16 0.374 

V17 0.287 

V18 0.743 

V19 0.197 

V20 0.446 

V21 0.538 

V23 0.930 

V24 0.196 

V26 0.738 

V27 0.356 

V29 0.731 

V30 0.788 

V31 0.830 


MODEL MODIFICATION INDICES 

Minimum M.I. value for printing the modification index 10.000 
M.I. E.P.C. StdE.P.C. StdYX E.P.C. 

No modification indices above the minimum value. 


TECHNICAL 1 OUTPUT 

PARAMETER SPECIFICATION 

LAMBDA 

CON PERS LESS MAIN ALIGN 
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V3 0 

V4 1 

V5 0 

V6 0 

V8 0 

V 12 0 

V13 0 

V 14 0 

V 16 0 

V17 0 

V18 0 

V19 0 

V20 0 

V21 0 

V23 0 

V24 0 

V26 0 

V27 0 

V29 0 

V30 0 

V31 0 


0 

0 

0 

2 

3 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


0 0 0 

0 0 0 

0 0 0 

0 0 0 

0 0 0 

0 0 0 

0 0 0 

0 0 0 

0 0 0 

7 0 0 

0 0 0 

8 0 0 

9 0 0 

0 0 10 

0 110 

0 0 0 

0 0 12 

0 0 13 

0 0 0 

0 0 14 

0 0 15 


LAMBDA 

INCLU 


V3 0 

V4 0 

V5 0 

V6 0 

V8 0 

V 12 4 

V13 5 

V 14 0 

V16 6 

V 17 0 

V18 0 

V19 0 

V20 0 

V21 0 

V23 0 

V24 0 

V26 0 

V27 0 

V29 0 

V30 0 

V31 0 


THETA 

V3 V4 V5 V6 V8 


V3 
V4 
V5 
V6 
V8 
V 12 


16 

0 

0 

0 

0 

0 


17 

0 

0 

0 

0 


18 

0 

0 

0 


19 

0 

0 


20 

0 


6/29/05 



CITES Survey Development and Validation 84 


V13 

0 

0 

0 

0 

0 

V 14 

0 

0 

0 

0 

0 

V 16 

0 

0 

0 

0 

0 

V17 

0 

0 

0 

0 

0 

V18 

0 

0 

0 

0 

0 

V19 

0 

0 

0 

0 

0 

V20 

0 

0 

0 

0 

0 

V21 

0 

0 

0 

0 

0 

V23 

0 

0 

0 

0 

0 

V24 

0 

0 

0 

0 

0 

V26 

0 

0 

0 

0 

0 

V27 

0 

0 

0 

0 

0 

V29 

0 

0 

0 

0 

0 

V30 

0 

0 

0 

0 

0 

V31 

0 

0 

0 

0 

0 


THETA 






V 12 

V13 

V14 

V16 

V 17 

V 12 

21 





V13 

0 

22 




V 14 

0 

0 

23 



V16 

0 

0 

0 

24 


V17 

0 

0 

0 

0 

25 

V18 

0 

0 

0 

0 

26 

V 19 

0 

0 

0 

0 

0 

V20 

0 

0 

0 

0 

0 

V21 

0 

0 

0 

0 

0 

V23 

0 

0 

0 

0 

0 

V24 

0 

0 

0 

0 

0 

V26 

0 

0 

0 

0 

0 

V27 

0 

0 

0 

0 

0 

V29 

0 

0 

0 

0 

0 

V30 

0 

0 

0 

0 

0 

V31 

0 

0 

0 

0 

0 


THETA 






V 18 

V19 

V20 

V21 

V23 

V18 

27 





V19 

0 

28 




V20 

0 

29 

30 



V21 

0 

0 

0 

31 


V23 

0 

0 

0 

0 

32 

V24 

0 

0 

0 

0 

0 

V26 

0 

0 

0 

0 

0 

V27 

0 

0 

0 

0 

0 

V29 

0 

0 

0 

0 

0 

V30 

0 

0 

0 

0 

0 

V31 

0 

0 

0 

0 

0 


THETA 






V24 

V26 

V27 

V29 

V30 


V24 33 

V26 0 34 
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V27 

0 

0 

35 



V29 

0 

0 

36 

37 

V30 

0 

0 

0 

0 


V31 

0 

0 

0 

0 


THETA 





V31 






V31 

39 





PSI 






CON 


PERS 

LESS 


MAIN 

CON 

40 





PERS 

41 

42 




LESS 

43 

44 

45 



MAIN 

46 

47 

48 


49 

ALIGN 

50 

51 

52 


53 

INCLU 

55 

56 

57 


58 


PSI 

INCLU 


INCLU 60 

STARTING VALUES 
LAMBDA 



CON 

PERS 

LESS 

MAIN 

V3 

1.000 

0.000 

0.000 

0.000 

V4 

1.000 

0.000 

0.000 

0.000 

V5 

0.000 

1.000 

0.000 

0.000 

V6 

0.000 

1.000 

0.000 

0.000 

V8 

0.000 

1.000 

0.000 

0.000 

V 12 

0.000 

0.000 

0.000 

0.000 

V13 

0.000 

0.000 

0.000 

0.000 

V 14 

0.000 

0.000 

0.000 

0.000 

V16 

0.000 

0.000 

0.000 

0.000 

V17 

0.000 

0.000 

1.000 

0.000 

V18 

0.000 

0.000 

1.000 

0.000 

V19 

0.000 

0.000 

1.000 

0.000 

V20 

0.000 

0.000 

1.000 

0.000 

V21 

0.000 

0.000 

0.000 

0.000 

V23 

0.000 

0.000 

0.000 

1.000 

V24 

0.000 

0.000 

0.000 

1.000 

V26 

0.000 

0.000 

0.000 

0.000 

V27 

0.000 

0.000 

0.000 

0.000 

V29 

0.000 

0.000 

0.000 

0.000 

V30 

0.000 

0.000 

0.000 

0.000 

V31 

0.000 

0.000 

0.000 

0.000 


LAMBDA 

INCLU 


V3 0.000 

V4 0.000 


ALIGN 


54 

59 


ALIGN 


0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

1.000 

0.000 

0.000 

1.000 

1.000 

1.000 

1.000 

1.000 
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V5 

0.000 

V6 

0.000 

V8 

0.000 

V 12 

1.000 

V13 

1.000 

V 14 

1.000 

V 16 

1.000 

V17 

0.000 

V18 

0.000 

V19 

0.000 

V20 

0.000 

V21 

0.000 

V23 

0.000 

V24 

0.000 

V26 

0.000 

V27 

0.000 

V29 

0.000 

V30 

0.000 

V31 

0.000 


THETA 



V3 

V4 

V5 V6 

V8 


V3 

0.329 





V4 

0.000 

0.348 




V5 

0.000 

0.000 

0.232 



V6 

0.000 

0.000 

0.000 

0.483 


V8 

0.000 

0.000 

0.000 

0.000 

0.177 

V 12 

0.000 

0.000 

0.000 

0.000 

0.000 

V13 

0.000 

0.000 

0.000 

0.000 

0.000 

V 14 

0.000 

0.000 

0.000 

0.000 

0.000 

V16 

0.000 

0.000 

0.000 

0.000 

0.000 

V17 

0.000 

0.000 

0.000 

0.000 

0.000 

V18 

0.000 

0.000 

0.000 

0.000 

0.000 

V19 

0.000 

0.000 

0.000 

0.000 

0.000 

V20 

0.000 

0.000 

0.000 

0.000 

0.000 

V21 

0.000 

0.000 

0.000 

0.000 

0.000 

V23 

0.000 

0.000 

0.000 

0.000 

0.000 

V24 

0.000 

0.000 

0.000 

0.000 

0.000 

V26 

0.000 

0.000 

0.000 

0.000 

0.000 

V27 

0.000 

0.000 

0.000 

0.000 

0.000 

V29 

0.000 

0.000 

0.000 

0.000 

0.000 

V30 

0.000 

0.000 

0.000 

0.000 

0.000 

V31 

0.000 

0.000 

0.000 

0.000 

0.000 


THETA 






V 12 

V13 

V14 V16 V17 

V 12 

0.367 





V13 

0.000 

0.276 




V 14 

0.000 

0.000 

0.256 



V 16 

0.000 

0.000 

0.000 

0.222 


V17 

0.000 

0.000 

0.000 

0.000 

0.275 

V18 

0.000 

0.000 

0.000 

0.000 

0.000 

V19 

0.000 

0.000 

0.000 

0.000 

0.000 

V20 

0.000 

0.000 

0.000 

0.000 

0.000 
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V21 

0.000 

0.000 

0.000 

0.000 

0.000 

V23 

0.000 

0.000 

0.000 

0.000 

0.000 

V24 

0.000 

0.000 

0.000 

0.000 

0.000 

V26 

0.000 

0.000 

0.000 

0.000 

0.000 

V27 

0.000 

0.000 

0.000 

0.000 

0.000 

V29 

0.000 

0.000 

0.000 

0.000 

0.000 

V30 

0.000 

0.000 

0.000 

0.000 

0.000 

V31 

0.000 

0.000 

0.000 

0.000 

0.000 



THETA 






V 18 

V19 

V20 

V21 

V23 

V18 

0.430 





V19 

0.000 

0.204 




V20 

0.000 

0.000 

0.282 



V21 

0.000 

0.000 

0.000 

0.555 


V23 

0.000 

0.000 

0.000 

0.000 

0.321 

V24 

0.000 

0.000 

0.000 

0.000 

0.000 

V26 

0.000 

0.000 

0.000 

0.000 

0.000 

V27 

0.000 

0.000 

0.000 

0.000 

0.000 

V29 

0.000 

0.000 

0.000 

0.000 

0.000 

V30 

0.000 

0.000 

0.000 

0.000 

0.000 

V31 

0.000 

0.000 

0.000 

0.000 

0.000 


THETA 






V24 

V26 

V27 

V29 

V30 

V24 

0.269 





V26 

0.000 

0.593 




V27 

0.000 

0.000 

0.323 



V29 

0.000 

0.000 

0.000 

0.492 


V30 

0.000 

0.000 

0.000 

0.000 

0.481 

V31 

0.000 

0.000 

0.000 

0.000 

0.000 


THETA 

V31 


V31 0.520 

PSI 



CON 

PERS 

LESS 

MAIN 

ALIGN 

CON 

0.050 





PERS 

0.000 

0.050 




LESS 

0.000 

0.000 

0.050 



MAIN 

0.000 

0.000 

0.000 

0.050 


ALIGN 

0.000 

0.000 

0.000 

0.000 

0.050 

INCLU 

0.000 

0.000 

0.000 

0.000 

0.000 


PSI 

INCLU 


INCLU 0.050 
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Appendix E: Revised Survey (Post Pilot Study) 


CITES Survey Revised May 31, 2005 

Note: Some questions were adapted from Woolfolk, A. & Hoy, W. (1990). Prospective teachers’ sense of efficacy 
and beliefs about control. Journal of Educational Psychology, 82(1), 81-91. 

1. Overall, I am comfortable teaching the curriculum in my assigned classes. 

strongly disagree 

disagree 

agree 

strongly agree 

2. I am well prepared to teach my assigned content classes. 

strongly disagree 

disagree 

agree 

strongly agree 

3. The amount a student can learn is primarily related to family background. 

strongly disagree 

disagree 

agree 

strongly agree 

4. If I try really hard, I can get through to even the most difficult or unmotivated students. 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 


5. When one of my students gets a better grade than he/she typically gets, it is usually 
because I found a better way of teaching that student. 

strongly disagree 

disagree 

agree 

strongly agree 

6. Some students in my classes would be better placed with a teacher who has additional 
training in meeting the needs of students with learning disabilities. 
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strongly disagree 

disagree 

agree 

strongly agree 

7. I need additional support (e.g., classroom aides) to address the needs of all students in 
my classes. 

strongly disagree 

disagree 

agree 

strongly agree 

8. Having students with special needs in my classroom has negative consequences for other 
students. 

strongly disagree 

disagree 

agree 

strongly agree 

9. For students with special needs to succeed, I have to take instructional time away from 
other students. 

strongly disagree 

disagree 

agree 

strongly agree 

10. The main ideas of the curriculum in my content area(s) are addressed across the units in 
my class: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 
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1 1. 1 review the ‘big picture’ of my curriculum to ensure that there is coherence across 
lessons in my class: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

12. I teach my students to look for the same main ideas across multiple lessons. 

strongly disagree 

disagree 

agree 

strongly agree 

13. My lessons are designed to explicitly support the broader goals within my curriculum: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

14. 1 modify the curriculum in my classes to match the state standards: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

15.1 use specific techniques (e.g., outlines, graphic organizers, note taking) to make the main 
ideas obvious in my class: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

16. 1 provide my students with multiple opportunities to practice using the main ideas of the 
curriculum in my content area: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 
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17. 1 primarily use the state standards to identify what is important to teach in my classes: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

18. My curriculum, instruction, and assessments are aligned to the main ideas in the state 
standards: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

19. 1 develop lessons plans to teach students the main ideas in the state standards: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

20. 1 plan assessments to measure student mastery of main ideas relative to the state 
standards: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

21. 1 use the state standards to identify what is important to assess in my classes: 

less than 25% of the time 
26% - 50% of the time 
51% - 75% of the time 
more than 75% of the time 

22. For students in my classes, improvement on state tests is extremely difficult to 
accomplish. 

strongly disagree 

disagree 

agree 

strongly agree 
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23. The state’s expectations are unreasonable for students at my school. 

strongly disagree 

disagree 

agree 

strongly agree 

24. 1 know how students at my school compare to students at other schools in my state on 
state tests. 

strongly disagree 

disagree 

agree 

strongly agree 

25. 1 know how students at my school compare to students across the nation in terms of their 
performance on standardized tests. 

strongly disagree 

disagree 

agree 

strongly agree 

26. State testing appropriately measures my students’ strengths and weaknesses. 

strongly disagree 

disagree 

agree 

strongly agree 

27. The statewide assessment addresses the main ideas of state content standards. 

strongly disagree 

disagree 

agree 

strongly agree 

28. Please estimate the percentage of your students who will fail the state test in spite of 
everything you do. 

[text response] 
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29. Please select the TWO answer choices below that you think have the most potential to 
improve student achievement. 

teacher staff development 
better curriculum planning 
improved materials selection 
include teachers in decisions 
mandate standardized curriculum 
let teachers control curriculum 
more frequent assessments 
other 

30. If you selected OTHER in the previous question, please describe the approaches that you 
think have the most potential to improve student achievement. 

[text response] 


Part II 

The following questions are included in order to give us demographic information that will 
aid in grouping responses for analysis. Your honest answers are very much appreciated. 

3 1 . What is your gender? 

male 

female 

32. What is your age? 

[text response] 

33. What is your race/ethnicity? 

[text response] 

34. What is the highest degree you have earned? 

BA/BS 

MA/MS/MEd 

DEd/PhD 

35. Including this year, how many years have you taught? 

[numeric response] 

36. What grade levels and content areas are you licensed to teach? 

[text response] 
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37. What content area(s) are you teaching this year? 

[text response] 

38. What grade level(s) are you teaching this year? 

[text response] 

39. Approximately how many students attend your school? 

fewer than 250 
251 -500 
501 - 1000 
1001 - 1500 
more than 1500 

40. Approximately what percentage of students at your school receive free/reduced price 
lunch? 

less than 10% 

11 % - 20 % 

21% - 30% 

31% -40% 

41% - 50% 
more than 50% 

41. How would you characterize your school’s setting? 
rural 

small town 

urban 

suburban 

42. Please feel free to share any additional thoughts you might have on topics related to 
education, (optional) 

[text response] 
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Abstract 


This technical document provides guidance to educators on the creation and interpretation of 
survey instruments, particularly as they relate to an analysis of program implementation. 
Illustrative examples are drawn from a survey of educators related to the use of the easyCBM 
learning system. This document includes specific sections on considerations related to sampling 
plans; instrument creation; and gathering, analyzing, and reporting of survey results. 
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SLDS Technical Document #4: 

Instrument and Survey Analysis Technical Report: Program Implementation Survey 

We begin this technical document with a brief definition of key terms and then provide 
more in-depth discussion of considerations related to sampling plans; instrument creation; and 
gathering, analyzing, and reporting of survey results. 

Key Terms 

Please note: These terms are defined in relation to surveys, here, but they may also have 
additional meaning when used in different contexts. Definitions provided here are not intended to 
be exhaustive. 

Instrument: A testing device for measuring a given phenomenon. Instruments may include 
paper/pencil or computer-based tests, interviews, and surveys, among others. 

Survey: A data collection tool used to gather infonnation from individuals. 

Respondent: A person responding to a survey. 

Population: The group of people whose input on a topic one is trying to obtain through 

surveying. If the population is small enough, it might be possible to include everyone in 
the population in one’s sample. Otherwise, one uses a sampling plan to identify in 
advance the people who will be included in the sample, and how they will be selected for 
inclusion. 

Sample: The group of people to whom the survey is sent. 

Selected Response Item: A question where respondents select their response from a pre- 
determined set of possible answer choices (aka “multiple choice”). 

Constructed Response Item: A question where respondents answer a question using their own 
wording, typically in a short-answer or essay-type fonnat (aka “open ended question”). 



Instrument & Survey Analysis 


Page 2 


Considerations in Sampling 

One of the first decisions one must make before conducting a survey is whom to include. 
Typically, this is done through the creation of a written sampling plan, whereby one specifies the 
population of interest and describes whether everyone in the population will be included 
{exhaustive or census sampling ) or a smaller sub-set of the population. In schools, it is not 
uncommon to solicit input from every member of the population of interest. In large districts, 
however, the sheer number of potential respondents may make a smaller sample more 
manageable. Knowing in advance how the results will be compiled and analyzed can help in 
making decisions about sampling plan. For instance, if one is going to be using a paper/pencil 
survey and hand-entering all the results prior to analysis, then a smaller, more selective, planned 
sample may be in order. In contrast, if one is using one of the many online survey tools where 
the results are tabulated by the computer as they come in, a larger sample may be feasible. 

If one decides to sample (rather than send a survey to every member of the population in 
question), the next decision is what sampling method to use. Generally speaking, sampling 
methods can take either a probability or nonprobability approach. 

Probability Sampling. In probability sampling, each member of the population has a 
chance of being selected, and one either samples randomly or systematically. In a random sample, 
each member of the population has an equal opportunity to be selected. In a systematic sample, 
one selects people to include in the survey using a systematic, planned approach. For instance, 
one might select every n th name to survey. One danger of this approach, of course, is that names 
are not randomly distributed across the population. Another systematic sampling technique 
stratified sampling first requires one to identify relevant clusters, or stratifications, into which 
members of the population might be grouped. Once these groups are identified, and the 
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population is organized by the stratifications, one selects a sufficient number of people from each 
of the groups to enable generalization to the population in question. In a school district this 
approach might be taken, for instance, if educators wanted to ensure that they had attempted to 
include input from a representative sample of people from each ethnic group enrolled in the 
school. 

Nonprab ability sampling. Nonprobability sampling involves selecting people to survey 
in a nonrandom manner. Often, one will see such sampling plans referred to as convenience, 
judgment, or snowball sampling. As the name implies, convenience sampling refers to a situation 
whereby people are selected to participate in the survey because they are convenient. For 
example, a school might conduct a survey of parents in attendance at back-to-school night using 
convenience sampling. One danger of convenience sampling, of course, is that the people 
included may not truly represent the larger population to which one is hoping to generalize. In 
the example of sampling parents at a back-to-school night, for instance, this approach might 
result in an inability to gather input from parents who work nights. To the extent that their views 
might differ from the views of the parents in attendance at the evening event, the sample results 
would be biased. Judgment sampling is typically an extension of convenience sampling whereby 
participants are selected based on the judgment of the person conducting the survey that they will 
be a good representation of the population as a whole. For instance, a school might opt to survey 
students in one classroom per grade, selecting those classrooms that appear to best represent the 
school population as a whole. The biggest concern with this approach, of course, is the degree to 
which the sample was selected wisely. To the extent that the sample differs from the population 
as a whole, the results may not be generalizable. 
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A final survey technique, most often used with hard-to-identify samples, is called 
snowball sampling. In this approach, one uses referrals from others to identify and recruit 
additional people to include in the sample. For instance, a school district may want to solicit 
information about how they might better provide assistance and support to homeless students and 
their families, but it is unlikely that school records will enable the easy and accurate 
identification of which parents should be surveyed. Using a snowball sampling technique, one 
would identify one or more homeless family in the district and would then request that person’s 
assistance in passing along the survey to other families in similar circumstances. A caution when 
using this approach is that the final sample may well be biased, as the people surveyed may end 
up sharing characteristics, through their associations with one another, other than the one that 
originally led to their identification. 

What to Look for in Reference to Sampling Design in Research. Because the approach 
used to gather the sample can impact the generalizability of the results considerably, it is 
important to examine the sampling plan and sample characteristics carefully when reading the 
results of a survey. Things to notice include how the sample was generated, the demographic 
characteristics of the sample (especially as they compare to your own population of interest), and 
the size of the sample compared to the identified population. Generally speaking, the closer the 
sample is in size and characteristics to the population as a whole, the more confident one can be 
in the results being reported. 

Considerations in Instrument Creation 

Surveys vary widely in their construction and intended use. Knowing in advance what 
one will do with the results can help ensure that the survey is useful and focused (Fowler, 2009). 
Keep in mind that if the survey is too long, it is likely that fewer people will complete it. 
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Generally speaking, responses to selected-response items are easier to analyze (many of the 
online survey tools provide a way to download results directly to an Excel file). Oftentimes, 
though, one may want to include constructed-response items or places where respondents may 
add more detail to explain their responses. Such responses can provide interesting insight into 
patterns in the results. It is important to keep in mind, however, the potential that a well-crafted 
constructed response might unduly sway one’s interpretation of the results. 

Creating selected-response items. In designing a survey, it is important to keep in mind 
that well- written questions will help respondents provide accurate and reliable answers. If your 
survey includes questions related to a sensitive topic, it may be wise to ensure that responses 
remain anonymous and to make sure that respondents are aware of this before they begin the 
survey. To avoid the situation where people select the ‘safe’ middle option choice rather than 
taking a stand on a topic with a more definitive answer, it is generally recommended to have an 
even number of response options when asking about people’s opinions on a topic. The following 
guidelines can help ensure that questions solicit useful responses: 


Table 1 


Selected-Response Item Writing Guidelines 


Guideline 

Example of a Poorly- Written 
Question 

Example of a Better- Written 
Question 

Limit each question to one 

How fair and appropriate are the 

How appropriate are the tests in this 

dimension 

tests in this class? 

class: 


a) extremely 

a) extremely 


b) somewhat 

b) somewhat 


c) not very 

c) not very 


d) not at all 

d) not at all 
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Table 1 


Selected-Response Item Writing Guidelines (Continued) 


Minimize the repetition across 
answer choices ~ include words that 
might repeat in the question stem so 
they do not need to be included in 
the responses. 

How would you describe the 
campus? 

a) I feel very safe on campus. 

b) I feel somewhat safe on 
campus. 

c) I do not feel very safe on 
campus 

d) I do not feel safe at all on 
campus. 

How safe do you feel on campus? 

a) very 

b) somewhat 

c) not very 

d) not at all 

Keep the grammar structure parallel 
in each of the possible response 
choices. 

How useful are the student behavior 
reports during parent conferences? 

a) best ever! 

b) they are quite useful 

c) somewhat useful 

d) not at all 

How useful are the student behavior 
reports during parent conferences? 

a) extremely 

b) quite 

c) somewhat 

d) not at all 

Make sure that the answer choices 
cover all possible responses. If 
people might feasibly be able to 
select two options, be sure they are 
given that choice. 

Which class do you most enjoy? 
(select one) 

a) English 

b) Mathematics 

c) Spanish 

d) PE 

Which class(es) do you most enjoy? 
(select all that apply) 

a) English 

b) Mathematics 

c) Spanish 

d) PE 

e) Other (explain) 

Make sure that the answer choices 
are mutually exclusive. If they are 
not, then respondents might not 
know how to give an accurate 
answer. 

Where do you usually do your 
homework? 

a) In the kitchen 

b) In front of the TV 

c) In my bedroom 

Where do you usually do your 
homework? 

a) In a quiet place at home 

b) In a noisy place at home 

c) In a quiet place away from 
home 

d) In a noisy place away from 
home 

e) Other (explain) 
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Table 1 

Selected-Response Item Writing Guidelines (Continued) 

Ensure that the questions flow 
smoothly from one to the next. 
Grouping questions that are similar 
can help make it easier for 
respondents to complete the survey. 

How long have you lived in the 
district? 

a) >5 years 

b) < or = 5 years 

What is your GPA 

a) 4.0 -3.5 

b) 3.49-3.0 

c) <3.0 

Where did you live before you lived 
here? 

a) in a neighboring district 

b) in a different state 

c) NA: I have always lived 
here 

How long have you lived in the 
district? 

a) >5 years 

b) < or = 5 years 

Where did you live before you lived 
here? 

a) in a neighboring district 

b) in a different state 

NA: I have always lived here 
What is your GPA 

a) 4.0 3.5 

b) 3.49-3.0 

c) <3.0 

If there is a chance that a respondent 
will not be able to answer a 
question, include a “don’t know” 
answer option. Otherwise, 
respondents may guess, reducing the 
reliability of your results. 

What proportion of your waking 
hours do you typically spend 
reading? 

a) Less than 25% 

b) More than 25% 

What proportion of your waking 
hours do you typically spend 
reading? 

a) Less than 25% 

b) More than 25% 

c) I don’t know 

Write questions and responses in 
such a way that they do not “lead” to 
a particular response. 

Do you agree that reading for 
pleasure outside of school is 
extremely important? 

a) Yes 

b) No 

c) I’m not sure 

How important is reading for 
pleasure outside of school? 

a) Extremely 

b) Somewhat 

c) Not very 

d) Not at all 

Avoid the use of unfamiliar words, 
abbreviations, and technical jargon. 

How satisfied are you with the 
district’s efforts to support work 
toward AYP? 

a) Extremely 

b) Somewhat 

c) Not very 

d) Not at all 

How satisfied are you with the 
district’s efforts to support 
improvement in student performance 
on statewide large-scale 
assessments? 

a) Extremely 

b) Somewhat 

c) Not at all 

d) I’m not sure 
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Table 1 


Selected-Response Item Writing Guidelines (Continued) 


Limit questions that ask respondents 

Rank the following in their order of 

Rank the following characteristics of 

to rank order a series of items to no 

importance: (1 = most important; 10 

cafeteria food in their order of 

more than 5 items. 

= least important) 

importance: (1 = most important; 5 = 


a) healthy cafeteria food 

least important) 


b) tasty cafeteria food 

a) healthy 


c) inexpensive cafeteria food 

b) tasty 


d) a variety of cafeteria food 

c) inexpensive 


e) large servings of cafeteria 

d) a variety 


food 

f) easily-recognizable 
cafeteria food 

g) cthnically-diverse cafeteria 
food 

h) warm cafeteria food 

i) etc. 

e) large servings 


Creating constructed-response items . A great deal of attention must be paid to the way 
in which constructed-response items are phrased if one is to get a lot of infonnation from them. 
A good rule of thumb to remember is that the best constructed-response questions will solicit 
more elaborate responses. Avoid phrasing a question in a way that might lead a respondent to 
answer with a single word response (eg., “yes” or “no”). Phrasing a question the same way you 
might if you were going to be conducting an interview can sometimes help. For instance, asking 
people to Describe the most useful aspects of today ’s professional development workshop may 
elicit more detailed responses than asking people What made today ’s professional development 
workshop useful? Likewise, one might decide to phrase a question in a neutral way that would 
enable respondents to share insights either negative or positive as a means of ascertaining their 
true feelings on a topic. For instance, a question phrased as Describe your opinion of today ’s 
professional development workshop could elicit either negative or positive responses, which 
might provide a more accurate gauge of how people viewed the event than responses that 
specifically solicit feedback about positive aspects of the workshop. 
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What to Look for in Reference to Item Type in Research. Often, survey designers will 
include a mix of selected- and constructed-response items on the survey instrument. It is 
important to remember, however, that not all respondents will have the same ability to express 
themselves well in writing. There is always the danger that a well-written constructed-response 
will skew one’s interpretation of the results as a whole. For this reason, it may be wise to use the 
results of selected-response items to help balance the opinions shared in open-ended answers. 
When reading the results of survey research, look for examples of the questions themselves and 
evaluate them for lack of bias and appropriate phrasing, keeping in mind that poorly- written 
questions reduce the reliability and validity of conclusions drawn from the survey. 
Considerations in Administering Surveys 

Coming up with a sampling plan and creating a survey instrument are critical first steps. 

It is also important, however, to make plans to increase the response rate. The following steps 
may help increase people’s willingness to respond to a survey. 

• If mailing the survey, use an envelope that will help your survey stand out in the mailbox. 
Generally speaking, colored, hand-addressed envelopes with postage stamps affixed 
(rather than bulk mail postage or printed address labels) may help reduce the chance that 
the survey is discarded as “junk mail” prior to being opened. 

• Include a brief but well- written cover letter, introducing the survey and emphasizing the 
importance of receiving a response from the person. If an online survey instrument is 
being used rather than a mailing, a short well-written e-mail takes the place of the printed 
cover letter. 


Include a title for your survey that is short and meaningful. 
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• Include short, easy-to-understand directions for completing and returning the 
survey. If the survey is being distributed through the mail, include the return 
address on the survey itself as well as on the reply envelope. 

• Begin the survey with a few interesting and non-threatening questions, and 
organize the survey such that the most important questions appear within the first 
half of the survey. That way, even if people do not complete the entire survey, the 
responses to the most important questions will be captured. 

• If the survey will be administered paper/pencil, leave plenty of white space. This 
helps give the impression that the survey will be easy to complete. 

• Consider incentives to encourage people to return completed surveys. For mailed 
surveys, attaching a dollar bill may help improve the response rate. Offering to send 
a summary report of the results to everyone who completes the survey may also 
help encourage people to respond. 

• Piloting the survey with a few people who represent the target population prior to 
distributing it more widely can help identify points of confusion to be corrected 
before the survey is administered more formally. 

Online versus Paper / Pencil Surveys. Widely-accessible technology in the fonn of 
internet-based survey tools has opened new avenues for collecting survey data. Online surveys 
may help reduce time and costs associated with gathering and analyzing survey data (Clayton & 
Werking, 1998; Gjestland, 1996; Schmidt, 1997). One of the most appealing characteristics of 
online survey engines is the way in which they facilitate data entry. Each respondent’s input is 
tallied as it is entered, enabling instantaneous download of the data for rapid analysis as soon as 
the survey is closed. They may also include built-in features to enhance response rates, such as 
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the ability to provide feedback and/or summaries of responses upon completion and simple-to- 
use tracking of who has responded, enabling the person conducting the survey to send out 
multiple invitations to participate without an increase in mailing costs (Dillman, 2000; Schmidt, 
1997), although some research suggests that paper/pencil surveys result in higher response rates 
(Sax, Gilmartin, & Bryant, 2003). Online survey engines are not without problems. One of the 
biggest challenges of using online survey engines is sampling bias that might favor higher- 
income households, where computer access might be more readily available (Sax et al., 2003). 

Paper/pencil surveys may have some distinct advantages for school-based applications, 
where potential respondents are likely to be physically present in a room (such as a parent or 
faculty meeting or student class). Distributing, completing, and collecting surveys in the same 
sitting, can improve response rates significantly. Likewise, some respondents may find a 
paper/pencil survey less intimidating to complete than one that is delivered via computer. 
Knowledge of the potential sample to be surveyed can help in selecting the most appropriate 
method to use. 

Analyzing Survey Results 

Survey results are generally reported for the whole sample as well as for specific sub- 
groups. To make such analyses possible, one must collect the data in such a way that it is 
possible to link the responses to the demographic information collected in earlier questions. For 
instance, if one is interested in comparing the opinions of freshmen students with the opinions of 
juniors, the survey instrument must include a question that will prompt students to select their 
class in school so that the responses can later be disaggregated or different surveys must be 
administered to students in the different groups. One of the most common mistakes made by 
people who have less experience with constructing and analyzing surveys is failure to include 
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questions that will enable thorough analysis of the results. Taking steps to ensure that all 
responses on a given survey can be tracked back to the individual responding to the survey does 
not mean breeching confidentiality. One can assign code numbers to each survey instrument and 
use the numbers to link individuals’ responses to different questions. Such an approach is 
particularly important for surveys that extend beyond a single sheet of paper; printing the 
person’s code number on each sheet of paper can help ensure that responses are tracked even if 
the individual pieces of paper become separated. 

Reporting Survey Results 

When reporting the results of a survey, provide a summary of the process used including 
the method of distribution and how the survey was introduced. Include infonnation about how 
surveys were distributed and how many were completed, calculating a response rate (usually 
reported as a %) for the survey as a whole. If some questions elicited significantly fewer 
responses than the overall completion rate for the survey, report this infonnation as well. Provide 
a frequency count, with raw numbers as well as percentage of respondents, for items that use a 
Likert-type scale. Including descriptive statistics (mean, standard deviation, and total number of 
respondents) may also help in interpreting results. Sometimes, other measures of central 
tendency such as which option elicited the most responses (mode) provide useful insight. If the 
survey includes constructed response items, one may either analyze the responses given for 
patterns and report findings with a few illustrative quotations or provide all responses given, 
allowing others to draw their own conclusions. Of these two approaches, the first may help 
reduce the potential for the qualitative responses to overpower the trends captured in the 
quantitative items on the survey. 
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Illustrative Example of Survey Creation, Administration, Analysis, and Reporting 

The section that follows provides an annotated technical report describing a survey 
created in the mid-2000s to collect information about teacher self-efficacy. It is included in its 
entirety, in the same form in which it was published, with one exception: the addition of 
annotations intended to call attention to features in the manuscript discussed in this technical 
document. 
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